1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
|
erofs-utils
===========
erofs-utils includes user-space tools for erofs filesystem images.
Currently only mkfs.erofs is available.
mkfs.erofs
----------
It can create 2 primary kinds of erofs images: (un)compressed.
- For compressed images, it's able to use several compression
algorithms, but lz4(hc) are only supported due to the current
linux kernel implementation.
- For uncompressed images, it can decide whether the last page of
a file should be inlined or not properly [1].
Dependencies
~~~~~~~~~~~~
lz4-1.8.0+ for lz4 enabled [2], lz4-1.9.3+ recommended [4].
How to build for lz4-1.9.0 or above
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To build you can run the following commands in order:
::
$ ./autogen.sh
$ ./configure
$ make
mkfs.erofs binary will be generated under mkfs folder. There are still
some issues which affect the stability of LZ4_compress_destSize()
* they have impacts on lz4 only rather than lz4HC * [3].
How to build for lz4-1.8.0~1.8.3
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For these old lz4 versions, lz4hc algorithm cannot be supported without
lz4 static libary due to LZ4_compress_HC_destSize unstable api usage,
which means only lz4 algrithm is available if lz4 static library isn't found.
On Fedora, static lz4 can be installed using:
yum install lz4-static.x86_64
However, it's not recommended to use those versions since there were bugs
in these compressors, see [2] [3] [4] as well.
How to generate erofs images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Currently lz4 and lz4hc are available for compression, e.g.
$ mkfs.erofs -zlz4hc foo.erofs.img foo/
Or leave all files uncompressed as a option:
$ mkfs.erofs foo.erofs.img foo/
How to generate legacy erofs images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Decompression inplace and compacted indexes have been introduced in
linux-5.3, which are not backward-compatible with older kernels.
In order to generate _legacy_ erofs images for old kernels,
add "-E legacy-compress" to the command line, e.g.
$ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/
Obsoleted erofs.mkfs
~~~~~~~~~~~~~~~~~~~~
There is an original erofs.mkfs version developped by Li Guifu,
which was replaced by the new erofs-utils implementation.
git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs
It may still be useful since new erofs-utils has not been widely used in
commercial products. However, if that happens, please report bug to us
as well.
Contribution
------------
erofs-utils is a GPLv2+ project as a part of erofs file system,
feel free to send patches or feedback to us.
To:
linux-erofs mailing list <linux-erofs@lists.ozlabs.org>
Li Guifu <bluce.liguifu@huawei.com>
Miao Xie <miaoxie@huawei.com>
Fang Wei <fangwei1@huawei.com>
Cc:
Gao Xiang <gaoxiang25@huawei.com>
Chao Yu <yuchao0@huawei.com>
Comments
--------
[1] According to the erofs on-disk format, the last page of files could
be inlined aggressively with its metadata in order to reduce the I/O
overhead and save the storage space.
[2] There was a bug until lz4-1.8.3, which can crash erofs-utils randomly.
Fortunately bugfix by our colleague Qiuyang Sun was merged in lz4-1.9.0.
For more details, please refer to
https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82
[3] There are many crash fixes merged to lz4 1.9.2 for LZ4_compress_destSize(),
and I once ran into some crashs due to those issues.
* Again lz4HC is not effected for this section. *
[LZ4_compress_destSize] Allow 2 more bytes of match length
https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda
[LZ4_compress_destSize] Fix rare data corruption bug
https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1
[LZ4_compress_destSize] Fix overflow condition
https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a
[LZ4_compress_destSize] Fix off-by-one error in fix
https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333
[LZ4_compress_destSize] Fix off-by-one error
https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba
since upstream lz4 doesn't have stable branch for old versions, it's
preferred to use latest upstream lz4 library (although some regressions
could happen since new features are also introduced to latest upstream
version as well) or backport all stable bugfixes to old stable versions,
e.g. our unoffical lz4 fork: https://github.com/erofs/lz4
[4] LZ4HC didn't compress long zeroed buffer properly with
LZ4_compress_HC_destSize()
https://github.com/lz4/lz4/issues/784
which has been resolved in
https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1
and already included in lz4-1.9.3, see:
https://github.com/lz4/lz4/releases/tag/v1.9.3
|
