Short: XPK based adaptive file compressor Author: racs@fs2.bdtf.hu (Almos Rajnai) Uploader: racs fs2 bdtf hu (Almos Rajnai) Type: util/pack Version: 0.11 Replaces: util/pack/headbanger.lha Requires: OS2.0, XPK package Architecture: m68k-amigaos Release date: 19-May-1999 --------------------------------------------------------------------- 1. What is Headbanger? This program is an XPK based adaptive file compressor utility, with only a command line interface. (It is not an archiver, but a one-file only compressor!) 2. How does it work? Everybody knows that the files are usually not a homogene byte-stream inside, a file almost always can be separated into some different types of data. (Think of a song module - it consists of a header, the song data, and the samples.) All kind of data can be compressed efficiently with a different kind of method. Fortunately Amiga has a great possibility of using different methods for different kind of files trough the great XPK package. If you have ever tried to compress eg. a module-file, you have probably realized that SQSH often achieves better ratio than even the famous LZX. (Which is known unbeatable in everyday archiving.) Well, I tought we have a lot of nice compressors at hand, but they are working only on one whole file only. How Headbanger works then (in short): - first it splits the file into small pieces, - it tries to apply all of the given XPK compressing methods (selecting from a list) on each pieces, - chooses the most efficient, and uses its data in the output. It gained more than 50% from large data files, unlike any other packers, archivers or compressors... (Even against RAR...) A testfile (it shows almost nothing about the efficiency but...): ChiMai.S3M music module (Scream Tracker 3 format, woth 16 bit wide samples) Original: 696964 bytes xpk gzip.100: 394040 bytes (43.5% won) lzx (strongest method, switch -9): 391920 bytes (43.7% won) lha 390779 bytes (43.9% won) xpk sqsh.100: 385092 bytes (44.7% won) xpk shri.100: 377584 bytes (45.8% won) headbanger, nibble:32Kb 351800 bytes (49.5% won) headbanger, nibble:16Kb 348312 bytes (50.0% won) headbanger, nibble:8Kb 344980 bytes (50.5% won) 3. How to use it? At first you need XPK package. Then a list of the methods to be used in ENV:, called "headbanger.methods". (Simply text file, I included one into this archive.) Why it is needed? Of course not all of the methods worth a try, and there are also methods (eg. IDEA), which are not compressors, but protectors. You can filter out these methods. (More methods, less speed...) This list-file is not really needed, but then only SHRI, SQSH and GZIP methods will be used. The format of this file is: one line is a method to be used, if it starts with * (asterisk), then before compressing the nibble a delta will be generated. (Useful for sample files and some others, this is how SQSH works, but it worth not using it on these kind of compressing methods: SQSH, SLZX, SMPL, SDHC.) An example: SQSH.100 GZIP.100 ELZX.100 *ELZX.100 The point, and the number represents the efficiency. (100 recommended of course.) Empty lines can be added (real empty lines only), and remarks beginning with the colon (;). Remarks and empty lines will be ignored. (See example file in this archive!) Then, all you have to know is command-line, parameters are: COMMAND/A,FILEIN/A,FILEOUT,THRESHOLD=THRE/K/N,NIBBLE/K/N,MAXNIBBLE=MAX/K/N COMMAND/A: required, can be: c - compress, d - decompress a file l - list the required methods for decompressing a compressed file, FILEIN/A: required, the name of the file to be processed (compress, decompress, or list) FILEOUT: not required, the name of the output file, if it is not given, then at compressing Headbanger adds a .hdbg suffix, at decompressing it tries to chop .hdbg suffix, or it adds .1 to the filename. THRESHOLD=THRE/K/N: not required, the minimum differences between the result of two methods, in case of changing method. Sometimes it worth keep the method of the previous nibble. That is because every new method requires a header in the packed file, thus increment output size. This number gives the size of minimum acceptable difference in number of bytes, when comparing the two output of the methods. If not given, it will be 20 bytes. This feature can be turned off by the amount of 0, but can not be negative of course. If you see alternating between two methods, you possibly have to raise this amount. Not advisable a much larger amount of 1% of the nibble size. NIBBLE/K/N: not required, the size of a nibble in bytes. If not given it will be 32768 bytes. Cannot be less than 256 bytes, and more than 65536 bytes. Usually 32 Kb or 16 Kb is the right amount, but can be different for each file. MAXNIBBLE=MAX/K/N: not required, determines the maximum size of a nibble, when Headbanger uses joined repacking. (See threshold.) Useful for lower memory machines. Be careful! If you have lot of memory, and you can use any size, on an other system depacking can fail because of low memory! If not given, then no limit used, else should be at least 2 times larger than nibblesize. 4. Worth it use? Well... The disadvantages of this tool are: - It is slooow... Depends on the methods you are using, but far slower than a usual archiver or compressor. - It is not fully automatic. You can play with the parameters, and cannot be sure that this is the optimal configuration. - It is rather an experiment, I see no reality of a Headbanger based archiver for example. (I included a batch anyway, called Archbanger, it uses LHA for compacting files, then uses Headbanger on it. Requires lot of space on the disk.) - It is not advised spread the compressed files, it is really annoying, when you are not able to decompress a file, because of a missing method... - It is a really beta release, not tested too deeply. - LZX sometimes gives a better ratio, I was not able to clue out the exact explanation of this. (Even when ELZX method is used...) Should be something with the headers or at the buffering. - There is no possibility of using compressed files in any other platform than Amiga. (No XPK ports.) - There is no check for corrupted files. (Some kind of CRC for example.) Advantages: - You can achieve a really fine compression ratio, especially for music modules and bitmap-type picture files. - When you get a new XPK method, you can immediately use it. - Decompression can be even faster than other compressors 5. Future plans I am planning now an XPK method based on the knowledge of designing Headbanger. Unfortunately I have some unanswered question yet, so it is not as simple as it seemed at first. 6. Legal This program is Freeware, except the sources. (Not included, I hate include my sources, I feel something like standing in the X-Ray...) Made with AmigaE v3.3a. Can be used in any non-commercial way you want, but please credit me! This package can be redistributed in any form, but the archive should be kept together, there is nothing to remove or add. If you have any question/remark/advice/request/bugreport, e-mail me: Rajnai Álmos (Álmos Rajnai is the english form of my name...) also known Rachy of BiřHazard racs@fs2.bdtf.hu or racs@deimos.bdtf.hu I am on #amigahu channel on IRCNet almost every weekdays. Sorry for my poor english, I will never learn it... 7. Thanks to Németh Zoltán (alias NemecheK), for correcting my bad english, and beta-testing Garami Ferenc, Bebe/AmigaRulez, qxy for beta-testing 8. History V0.11, 18-May-1999 *Fix: added XPK_CHUNKSIZE to the recompress of joined nibbles, it gives a bit more efficiency V0.1, 12-May-1999 First Aminet release *Fixes: bad english in docs (well...) better format support for methods file (empty lines and remarks) *Added: MAXNIBBLE parameter V0.1ß, 18-Apr-1999 First beta release