summaryrefslogtreecommitdiff
path: root/payload_generator
AgeCommit message (Collapse)Author
2021-09-14Consider .capex files as zip archivesKelvin Zhang
This reduces size of SP2A.210827.001-to-SP2A.210903.001.zip OTA package from 83MB to 66MB. Which is ~21% improvement. Tests: th Bug: 199920825 (cherry picked from commit dc1f258e94b2c2dac1d3468301ebcdc0a2fe147c) Change-Id: I10fa03401e5920c633c3ce5d6cf737489fecb6de Merged-In: I10fa03401e5920c633c3ce5d6cf737489fecb6de
2021-04-30Add COW version to the update_metadata.protoAkilesh Kailash
COW versioning will be used to detect if the COW library version matches with the version in the proto file. If not, VABC is disabled. Bug: 183863613 Test: Apply OTA and verifiy if VABC is disabled if the versioning doesn't match Signed-off-by: Akilesh Kailash <akailash@google.com> Change-Id: I2d8690bd30d3436c6eb281d3e8d6545cb4888a75
2021-04-20Refactor extent writer to take filedescriptor in constructorKelvin Zhang
Functions which receive an instance of extent writer need to manually pass fd to ExtentWriter via Init() call, which breaks separation of concerns. It makes it hard for us to decouple InstallOp execution from writing of data, as the execution unit must be aware of which fd to pass to extent writer. In addition, many extents writer, such as snapshot extent writer, simply ignores the fd parameter, which is a indication of poor code structure. To address the above issue, we pass FileDescriptorPtr via constructor if needed. This way, whoever is "executing" InstallOps don't need to care about where the output data is going, and whoever's writing the data would be responsible for initializing an ExtentWriter. Test: th Change-Id: I6d1eabde085eefd55da9ecc0352d4a16ae458698
2021-03-28Support verity writes in VABCKelvin Zhang
Test: generate an OTA with verity enabled, install it Bug: 173432386 Change-Id: I14cccb1bb339c9824a95b8e42ac5144cb8b75c3f
2021-03-10Allow disable_vabc flag in full OTAKelvin Zhang
Previously, brillo_update_payload only honors --disable_vabc flag for incremental OTA. Respect this flag for full OTA as well. Test: Generate a full OTA with --disable_vabc Change-Id: Ic18dfdffd3e43ab139d1ea0704679ea38dd082c2
2021-03-04Call CowWriter::Finalize() before attempting to get size of COWKelvin Zhang
When actually applying an OTA, we call Finalize() on each partition. Emulate this behavior on host side simulation tools. Test: th Change-Id: I68be3a450d68d446a42e0bf40c004375871675e0
2021-03-01Add a util for converting ota payload to cow imageKelvin Zhang
VAB team can use this tool to understand what real world COW images look like, and perhaps write unittests How to use this tool: 1. mm -j cow_converter 2. `unzip your_ota.zip payload.bin` 3. `unzip target_file.zip "IMAGES/*" -d /tmp/target_file 4. `cow_converter payload.bin /tmp/target_file/IMAGES` 5. inspect generated cow images `ls -l /tmp/target_file/IMAGES/*.cow` Test: cow_converter payload.bin /tmp/target_file/IMAGES Change-Id: I2c9d1ea566378e3350a048c40ff05a79d14873e8
2021-03-01Add a library function for performing a cow dry runKelvin Zhang
During OTA generation, delta_generator will perform a dry run of VABC OTA by convert install ops to cow ops and apply them to a cow writer. This code can be re-used by cow_converter, which is a util for converting OTA payload to COW image. Since the conversion happens at generation time, we will have access to both the source and target images. We can just read data from target image and write to the cow, no need to actually do bsdiff/decompression/puffdiff. Therefore we add some utility functions for writing COW with target image, these functions operate under different assumptions than update_engine on android devices. Test: th Change-Id: I9ab91918c8ee6773bdab04ecbf654e33b7a7d827
2021-02-22Add vabc_compression parameter to update_metadataKelvin Zhang
Test: treehugger Change-Id: Icc65db9a7c4fbdf22376f9900016ae1b7058c611
2021-02-11Reland: Pass apex_info.pb file to delta_generatorKelvin Zhang
target_files contain META/apex_info.pb, which contains metadata about compressed apexes. Extract this file from target_file.zip, and pass it to delta_generator. delta_generator will then copy these data to update_metadata Test: generate an OTA, make sure a device running an older build can install the OTA Bug: 172911822 Change-Id: If0e185a32262a849d533c3316ffdf205cb6628b6
2021-02-09Add a flag to manually override VABCKelvin Zhang
Test: generate an OTA with --disable_vabc Change-Id: I4a42319584dc1388d097dba343fe3ace82479bec
2021-02-02Skip merge sequence and COW computation if VABC disabledKelvin Zhang
Test: treehugger Bug: 177936022 Change-Id: I2d9e75f9c1818aa82c993b248c94b23696ea8426
2021-01-30Revert "Pass apex_info.pb file to delta_generator"Tianjie Xu
This reverts commit 4e5da3e9d7459e3d983db943ebf3cb4c36af9c60. Reason for revert: b/178908708 Change-Id: I616da64ffda5b94044f3493861d6d01aebfaad61
2021-01-28Pass apex_info.pb file to delta_generatorKelvin Zhang
target_files contain META/apex_info.pb, which contains metadata about compressed apexes. Extract this file from target_file.zip, and pass it to delta_generator. delta_generator will then copy these data to update_metadata Test: generate an OTA Bug: 172911822 Change-Id: Ia4babb9e711a92f2b78ef8e1dd6ad35d0a2bb5a8
2021-01-20Add a "vabc_enabled" flag to update_metadataKelvin Zhang
When doing a downgrade build, the target build might not support vabc. So disable vabc in this case. Test: treehugger Change-Id: Ie5a56adf99b495a40d1caafbda6466181da1bc10
2020-12-13Always write SOURCE_COPY blocks in reverse orderKelvin Zhang
Test: treehugger Bug: 174112589 Change-Id: If95893569ab41d1806f266aa269722b403a50fa4
2020-12-10Avoid self-overpping SOURCE_COPY ops by split themKelvin Zhang
snapuserd doesn't like self-overlapping copy operations. So we need to split these operations. See linked bugs for examples. Bug: 175137108 Test: treehugger Change-Id: I24dca63ae89849330561fc26d9f2038982ed7ef2
2020-12-05Merge remote-tracking branch 'aosp/upstream-master' into mergeKelvin Zhang
Test: treehugger Change-Id: I4984f03fa95a753fb17779451eb458f177432d4f
2020-11-23Add --disable_verity_computation flag to delta_generatorKelvin Zhang
VABC doesn't support writing hash tree yet, provide an option to disable for now Test: treehugger Bug: 168554689 Change-Id: I7bbb76b33b46b44101fc3ca6cee2c9ee85e2e82d
2020-11-17Cleanup PayloadSigner::AddSignatureToManifest()Sen Jiang
Change-Id: Idbde802b06f18fb26160586b7fe34c104c1b0bf9
2020-11-16Add `full_boot` option to brillo_update_payloadKelvin Zhang
Test: generate && apply an OTA with --full_boot Bug: 171906589 Change-Id: I773e808ba5914e5fcc1c7be9095fdbee7d7ebd54
2020-11-14update_engine: Make SystemState accessible from everywhereAmin Hassani
SystemState is supposed to be a global context and is used lamost everywhere. So instead of passing it to functions and keeping multiple pointers to it, its better to do what we did in dlcservice and make it a singleton class with a getter that can be get from everywhere. BUG=b:171829801 TEST=unittests Change-Id: I3b2de9394b7769b3911195ca52d61dbe49afd4dd Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2521792 Commit-Queue: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Jae Hoon Kim <kimjae@chromium.org>
2020-11-13Add small testcase for merge sequence generatorKelvin Zhang
Test: treehugger Change-Id: Ibea610fcc0d166a44f9425c9fc2e6d17cf877322
2020-11-09Rename PReadAll/PWriteAll for file descriptor ptr to ReadAll/WriteAllKelvin Zhang
Utils.cc contain 2 implementation of PReadAll: 1 for unix fd, and 1 for FileDescriptorPtr. However, the implementation for unix fd calls pread syscall under the hood, which does not change file offset. The implementation for FileDescriptorPtr DOES change file offset, making code inconsistent. For now we rename inconsistent functions to ReadAll/WriteAll. The next CL adds PReadAll/PWriteAll implementation for FileDescriptorPtr. Test: treehugger Change-Id: I2781b294f0f8e866275e1649e9b45d565d4cd5b8
2020-11-09Estimate COW image size during OTA generationKelvin Zhang
Estimate COW image size and put the estimation in OTA metadata. Then VAB could use this to allocate disk space and prompt the user if more space required. Test: create an OTA package Change-Id: Iaedafcf39af2d1a4d9cae9cd1a642a3cd3a4815c
2020-11-04Merge "Merge remote-tracking branch 'aosp/upstream-master' into merge"Tianjie Xu
2020-11-04Support host simulation of partial updateTianjie
There were two problems for partial update's host verification: 1. We cannot generate copy commands for partitions not included in the payload on host side. 2. Partial update should always be delta updates, while the logic was missing on host side. Address both issues in this cl. Bug: 171519321 Test: delta_generator --is_partial_update true --in_file=payload.bin \ --partition_names=product:system:system_ext:vbmeta_system \ --new_partitions=/tmp/1:/tmp/2:/tmp/3:/tmp/4 --major_version=2 Change-Id: I87c1b162d1688c04be0dd81566966eced3690ca1
2020-11-03Merge remote-tracking branch 'aosp/upstream-master' into mergeTianjie
git merge aosp/upstream-master --commit -s recursive Test: treehugger Change-Id: Ifab9f47e1c5bea3898e78df0139d10842b41a44f
2020-11-03update_engine: Create cros vs. aosp boundary clearAmin Hassani
Its time to make the boundary between Chrome OS and Android code more clear. This CL moves all CrOS only code to "chromeos" directory and the same for Android (in "android" directory). This way we would easily know which code is uses in which project and can keep the code cleaner and more maintainable. One big remaining problem is download_action* files. It seems like DownloadAction class does a lot of things that chrome OS needs and it depends on a lot of Chrome OS stuff, but Android is also using thie Action in a way that circumvent the Chrome OS stuff. For example Android checks for SystemState to be nullptr to not do things. This is really fragile and needs to change. Probably Android Team has to implement their own DownloadAction of some sort and not re use the Chrome OS one in a very fragile way. Removed a few android files that have not been used anywhere. Changed some clang-format and lint issues in order to pass preupload. BUG=b:171829801 TEST=cros_workon_make --board reef --test update_engine Change-Id: I3fff1d4a100a065a5c1484a845241b5521614d9f Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2508965 Tested-by: Amin Hassani <ahassani@chromium.org> Auto-Submit: Amin Hassani <ahassani@chromium.org> Reviewed-by: Jae Hoon Kim <kimjae@chromium.org> Reviewed-by: Tianjie Xu <xunchang@google.com> Reviewed-by: Kelvin Zhang <zhangkelvin@google.com> Commit-Queue: Amin Hassani <ahassani@chromium.org>
2020-11-02Fix hang on host apply payloadYifan Hong
When applying some payload binaries, delta_generator might hang because before the loop runs, StartProcessing already finishes, thus BreakLoop does nothing. Test: extract GKI from APEX finishes and does not hang Bug: 171519321 Change-Id: Iad9c12402d23ce2e34f15db553445ea134f2a36d
2020-10-29Merge remote-tracking branch 'remotes/aosp/upstream-master' into merge-crosAmin Hassani
Merge back the recent update_engine changes back to Android. Created by: $ git merge remotes/aosp/upstream-master --commit -s recursive No special conflict to resolve. Bug: 163153182 Test: None Change-Id: I4c65eb9c57448847857e2339935a5d47c8cb690a
2020-10-29Revert "update_engine: Remove case conditions for deprecated operations"Amin Hassani
This reverts commit f4d1196edca97a666e20d16a334aa67f47bbcf0a. Reason for revert: <Failing in aops/1479157> Original change's description: > update_engine: Remove case conditions for deprecated operations > > MOVE and BSDIFF were related to minor version 1 and major version 1 > which both are deprecated so we should remove these so the builders > don't complain (with warning) about these values. > > BUG=None > TEST=unittests pass > > Change-Id: I7ccc2c18d2dfc8e80b7c5d560988762a4c4cbdc3 > Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2265160 > Tested-by: Amin Hassani <ahassani@chromium.org> > Reviewed-by: Jae Hoon Kim <kimjae@chromium.org> > Commit-Queue: Amin Hassani <ahassani@chromium.org> TBR=ahassani@chromium.org,kimjae@chromium.org # Not skipping CQ checks because original CL landed > 1 day ago. Bug: None Change-Id: I9de3451f1607301e2b273a815e116754e2fa1e25 Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2506472 Reviewed-by: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Commit-Queue: Amin Hassani <ahassani@chromium.org>
2020-10-29update_engine: Fix leaking unit testsAmin Hassani
Some of the unit tests have been leaking temp files because they don't properly unlink them. In this CL, we did some rearrangement of the ScopedTempFile class and moved it into the utils.h (instead of testing only location) so it can be used everywhere and more efficiently. Also added functionality to open an file descriptor too so users don't have to keep a different object for the file descriptor. BUG=b:162766400 TEST=cros_workon_make --board reef --test; Then looked at the /build/reef/tmp directory and no files were leaked. Change-Id: Id64a2923d30f27628120497fdefe16bf65fa3fb0 Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2500772 Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Jae Hoon Kim <kimjae@chromium.org> Commit-Queue: Amin Hassani <ahassani@chromium.org>
2020-10-27Implement common library for converting InstallOps to Cow OpsKelvin Zhang
Test: generate && serve an OTA Bug: 168554689 Change-Id: If9e87ca3e993372ebb6b24ed64e71b319630bb18
2020-10-11update engine: clang formattingVyshu
Fix clang formatting issues for all .cc, .h, and .proto files in update engine. ~/trunk/src/chromium/src/buildtools/linux64/clang-format -i \ -style=file $(find update_engine -name '*.h' -o -name '*.cc' \ -o -name '*.cpp' -o -name '*.c') BUG=b:169679497 TEST=CQ pass Change-Id: I4fde01d3e734dbffaa2c7e7b667503d310abccae Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2462840 Reviewed-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Jae Hoon Kim <kimjae@chromium.org> Tested-by: Vyshu Khota <vyshu@google.com> Commit-Queue: Vyshu Khota <vyshu@google.com>
2020-09-30update_engine: Switch back crypto function calls to get0 versionAmin Hassani
Because of b/158580694 we had to switch the crypto calls to get1 version and manually release them. Since that bug has been marked as fixed, we can now switch it back to its original form. BUG=b:163153182 TEST=FEATURES=test emerge update_engine Change-Id: I8c2ff6619f592fc5e78a45efce14d42626d66034 Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2438992 Tested-by: Amin Hassani <ahassani@chromium.org> Auto-Submit: Amin Hassani <ahassani@chromium.org> Reviewed-by: Jae Hoon Kim <kimjae@chromium.org> Commit-Queue: Amin Hassani <ahassani@chromium.org>
2020-09-26Merge "update_engine: Merge remote-tracking branch 'cros/upstream' into ↵Commit Bot
cros/master"
2020-09-24payload_generator: Remove unused attributesVyshu
Some attributes are not used by the client. Removed all instances of of "ignored" and option ttributes from delta generation. Removed all instances of ImageInfo and moved target and source version to paygen_payload. BUG=b:163048638 TEST=FEATURES=test emerge-hatch update_engine TEST=FEATURES=test emerge update_payload TEST=./run_pytest lib/paygen/paygen_payload_lib_unittest.py Change-Id: I9102d37fcf054f2cbeb79e54113925a684de1cfb Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2388163 Reviewed-by: Amin Hassani <ahassani@chromium.org> Tested-by: Vyshu Khota <vyshu@google.com> Commit-Queue: Vyshu Khota <vyshu@google.com>
2020-09-17update_engine: Merge remote-tracking branch 'cros/upstream' into cros/masterAmin Hassani
Done with: git merge cros/upstream --commit -s recursive - Added EC key support and its unittests. - Resolved a conlict on error codes. Since Android versions are not uploading any UMA metrics, I gave the priority to the Android version Since they can't be changed. - Changed the openssl functions to get1 version (from get0) version because of a current issue with gale. Once the issue is resolved we need to change them back. - Some remaining styling issues fixed by clang-format BUG=b:163153182 TEST=CQ passes TEST=unittests Change-Id: Ib95034422b92433ce26e28336bc4806b34910d38
2020-08-31Implement the topology sort in CreateCowMergeOperationTianjie
Generate a sequence of COW_COPY operations. The extents in these operations won't be written as raw bytes in snapshots. Instead, they will read from the source partitions. So it's important to make sure no read after write happens on the source partitions, similiar to the inplace update. If the topology sort isn't possible due to cycles, we will omit some blocks to break the cycles. And these blocks will be carried as raw bytes in cow instead. Bug: 162274240 Test: generate a payload, unittests pass Change-Id: If4866704459b919d4bc09be48286b3e321b70497
2020-08-28Implement the functions in CowMergeOperationsTianjie
Implement the function to create an object & validate sequence. Bug: 162274240 Test: unit tests pass Change-Id: Id41460a886d94a98e154b222c3401a5f95b9e047
2020-08-27Add CowMergeOperations as a hint for snapshot writeTianjie
As proposed in http://go/vabc, we want to reduce the cow size for VAB. One nature apporach is to skip writing the idential blocks to snapshot; instead we can read from the souce blocks. Similiar to the non-A/B update schema, we need to compute a sequence for snapshot merge to avoid the read after write problem. If there is a circular dependency, we will omit some blocks in the result sequence to break the cycles. So libsnapshot will write the raw data of these blocks to cow. All extents in the CowMergeOperations are subsets of a particular OTA SOURCE_COPY InstallOperation. Also, these src & ext extents will be contiguous to improve the libsnapshot read performance before merge completes, as well as to simplify the sequence generation. Bug: 162274240 Test: unittest pass, genertes an OTA Change-Id: I12c952593d83a8e34a0a6cff5a2066c9103a0d30
2020-08-27Set per-partition timestamps in OTA generationKelvin Zhang
update_engine can also accept payload with per-partition timestamps. This CL updates OTA generation script to emit per-partition timestamps when writing an OTA package. Test: Generate && serve an ota Change-Id: I17529a004b8e0bbcb7d69dde93fb0fd7124b3b17
2020-08-18Fix race condition in blob_file_writerKelvin Zhang
SetTotalBlobs modifies total_blobs, so it chould grab a mutex first. Test: run delta_generator repeatly(~500 times) Change-Id: Ic0e3ab0298dee9a30c0f8ba414d506e10e3654ca
2020-07-29Update language to comply with Android's inclusive language guidanceTianjie
More details in: https://source.android.com/setup/contribute/respectful-code Bug: 161896447 Test: build, run unittests Change-Id: I6a7136b01ecce948a3997c60b7dcec848331e8ef
2020-07-29Update language to comply with Android's inclusive language guidanceTianjie
More details in: https://source.android.com/setup/contribute/respectful-code Bug: 161896447 Test: build, run unittests Change-Id: I8d666eee75490146eb57a183f0cfdf343b58b602
2020-07-29update_engine: Fix non inclusive vocabularyAndrew
Bug: 161896447 Test: build (cherry picked from commit cc6ab9f076694a816fa35f133e98e7737542ddd8) Change-Id: Ic321806ab6029c88723c220f243e2c2c7a9e94f0 Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2267058 Tested-by: Andrew Lassalle <andrewlassalle@chromium.org> Auto-Submit: Andrew Lassalle <andrewlassalle@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org> Commit-Queue: Amin Hassani <ahassani@chromium.org>
2020-07-29Parallelize delta generation across partitionKelvin Zhang
On my machine, this change alone reduces incremental OTA generation time from 56 minutes to 29 minutes Test: Generate and serve an OTA Change-Id: Id4ffc6f02f28594eb60cb934777b82f1899bbbc2
2020-07-29Update language to comply with Android's inclusive language guidanceSaint Chou
See https://source.android.com/setup/contribute/respectful-code for reference #inclusivefixit Bug: 161896447 Change-Id: I28df0e1a45fb625fd70b5cd471e01e7f60fcd652 Test: NA (Comment only)
2020-07-20Support generation of partial updatesTianjie
Add a new minor version kPartialUpdateMinorPayloadVersion for partial updates. Also, we always treat the partial update as a delta update in payload consumer, so new update_engine can perform minor version check correctly. Conceptually, partial update is indeed a delta update; because we need to copy | use the untouched partitions. Since the payload for the partial update doesn't carry old partition info, old update engines will treat them as full update. So old UE will also fail the minor version check correctly; because we always expect kFullPayloadMinorVersion for full updates. Bug: 157778739 Test: generate & apply partial full|incremental updates, generate regular updates, unittests pass Change-Id: I7f8365cf99098269150dd08e028120354944f3c6
generated by cgit v1.2.3 (git 2.25.1) at 2025-10-03 12:59:30 +0000