summaryrefslogtreecommitdiff
path: root/scripts/update_payload/payload.py
AgeCommit message (Collapse)Author
2021-03-11Improve load speed for python payload scriptKelvin Zhang
When using update_payload module to read a full OTA, it will attempt to load the entire payload into memory first, which is slow. Use mmap instead, as parsing manifest doesn't require the entire payload to be in memory, only the first couple of MB. In case of opening a zipfile, use zfp.open(), which returns a streaming interface. Test: import update_payload, load an OTA Change-Id: I3f7c7515dbf5576ccfda8d844fe22e9be34fdaed
2020-12-14Add host side simulation python scriptKelvin Zhang
This enables host side simulation of partial updates with the original complete target file Bug: 171519321 Test: run this script with a partial update python3 simulate_ota.py --source ~/aosp/aosp_cf_x86_phone-target_files-6949164.zip --target ~/aosp/aosp_cf_x86_phone-target_files-6937553.zip ~/aosp/cf_x86_downgrade.zip Change-Id: Iec52d8ac585b499d50bdb0c89d203ea353f2e94f
2020-09-26Merge "update_engine: Merge remote-tracking branch 'cros/upstream' into ↵Commit Bot
cros/master"
2020-09-24payload_generator: Remove unused attributesVyshu
Some attributes are not used by the client. Removed all instances of of "ignored" and option ttributes from delta generation. Removed all instances of ImageInfo and moved target and source version to paygen_payload. BUG=b:163048638 TEST=FEATURES=test emerge-hatch update_engine TEST=FEATURES=test emerge update_payload TEST=./run_pytest lib/paygen/paygen_payload_lib_unittest.py Change-Id: I9102d37fcf054f2cbeb79e54113925a684de1cfb Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/2388163 Reviewed-by: Amin Hassani <ahassani@chromium.org> Tested-by: Vyshu Khota <vyshu@google.com> Commit-Queue: Vyshu Khota <vyshu@google.com>
2020-08-19Update payload.py to work with OTA packages directlyKelvin Zhang
Test: python3 payload_info.py some_ota.zip Change-Id: Id688e87d4712f2f750320eb0cd86c6a2cc263313
2019-11-19update_payload: Port scripts to python3Andrew Lassalle
Update the update_payload scripts to be compatible with python2 and python3. Python2 compatibility is needed since the repo is shared with Android. BUG=chromium:1011631 TEST=Executed aosp/system/update_engine/scripts/run_unittests and cros_generate_update_payload Cq-Depend: chromium:1904837, chromium:1911499 Change-Id: Ie450b80b5f7550051b38d320173ccc0c915f65e7 Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/1904310 Commit-Queue: Andrew Lassalle <andrewlassalle@chromium.org> Tested-by: Andrew Lassalle <andrewlassalle@chromium.org> Reviewed-by: Mike Frysinger <vapier@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org> Auto-Submit: Andrew Lassalle <andrewlassalle@chromium.org>
2019-10-29update_engine: Deprecate major version 1Amin Hassani
We have moved away from major version 1 in Chrome OS and already have a stepping stone for it in M53. So this cleanup makes the code much easier to understand. BUG=chromium:1008553 TEST=FEATURES="test" sudo emerge update_engine update_payload TEST=cros_generate_update_payload --image chromiumos_test_image.bin --check --output delta.bin Change-Id: I01815dfa5fdf395f8214ef162e01ecca2d42f7fc Reviewed-on: https://chromium-review.googlesource.com/c/aosp/platform/system/update_engine/+/1857459 Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Sen Jiang <senj@chromium.org> Commit-Queue: Amin Hassani <ahassani@chromium.org>
2018-06-25update_payload: Allow specifying partition options for major version 2Tudor Brindus
This commit adds the ability to specify partition options for more than just kernel/rootfs. This supersedes -p/--root-part-size, -P/--kern-part-size, --dst_kern, --dst_root, --src_kern, --src_root, --out_dst_kern, and --out_dst_root. They are replaced by --part_names used in conjunction with --part_sizes, --dst_part_paths, --src_part_paths, and --out_dst_part_paths. Backwards-compatibility with the old flags is kept, so long as they are not used alongside the new flags. BUG=b:794404 TEST=no errors during run_unittests and test_paycheck.sh Change-Id: Icc1118abbf89dd268be3eafe41723657c5178197 Reviewed-on: https://chromium-review.googlesource.com/1103063 Commit-Ready: Tudor Brindus <tbrindus@chromium.org> Tested-by: Tudor Brindus <tbrindus@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org>
2018-03-23update_payload: Allow check for given metadata sizeAmin Hassani
Allow passing metadata size to check_update_payload so we can verify the metadata size in omaha equals to the one in the payload. BUG=chromium:820243 TEST=run paycheck.py with both valid and invalid metadata sizes reports as expected TEST=unittests Change-Id: Ib41ce77af77636fffec6752201c363e7fbbf868d Reviewed-on: https://chromium-review.googlesource.com/955679 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org>
2018-03-09Merge remote-tracking branch 'aosp/upstream-master'.Sen Jiang
Bug: 30989362 Test: update_engine_unittests Change-Id: I04417ac21508f4da3ead29b1dea34886f2fc84af Merged-In: I04417ac21508f4da3ead29b1dea34886f2fc84af
2018-01-30update_payload: migrate the license headers to AOSPAmin Hassani
update_engine/scripts/* have CROS license but the rest of UE have android license. This cl just migrates all CROS licenses to AOSP. BUG=None TEST=repo upload did not complain about the mismatched licenses. Change-Id: Id947be3e860baf7a523163965336f47d268c6808 Reviewed-on: https://chromium-review.googlesource.com/891688 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Amin Hassani <ahassani@chromium.org>
2018-01-29update_payload: remove block tracerAmin Hassani
block tracing was useful for minor version 1 with move operations (before having A/B updates). But, we do not create that operation anymore and we are in the process of moving to major version 2. So this needs to go BUG=chromium:794404 TEST=unit tests Change-Id: I56d24ef81e8f37c481669d47e5878eb3a321131f Reviewed-on: https://chromium-review.googlesource.com/888546 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org>
2018-01-11Merge remote-tracking branch 'goog/upstream-master'.Sen Jiang
The following commits were reverted: 840703a Fix update over cellular network on guest account eaad5d0 Do not merge to AOSP: Fixes the link to brillo-clang-format in CrOS 740efad Reboot even if a system update is not available. Fixed a few sign compare warnings. Had to ifdef out 2 SquashfsFilesystemTest because it depends on unsquashfs -m. Test: update_engine_unittests Change-Id: I6f4ca5003e78c76064ec60d0797505d8c18d00bf Merged-In: I6f4ca5003e78c76064ec60d0797505d8c18d00bf
2018-01-05update_payload: Fix most of lint styling issues.Amin Hassani
This patch fixes a lot of pylint issues in the update_engine scripts. Majority of this changes are based on recommendation found in: https://www.chromium.org/chromium-os/python-style-guidelines It is a good idea to do these changes now, because if there are many pylint errors when performing 'repo upload', serious problems can be overshadowed by a lot of noise and eventually cause problems. These fixes include: - Fixing executable shebangs to /usr/bin/python2. - Fixing import-error problems by disabiling them. - Removing pylint disables that are not valid anymore. - Changing all imports to proper absolute import format. - Change the import of PayloadError from update_payload.PayloadError for simplicity. - Add pydoc strings for functions and classes that were missing. The remaining unchanged pylint problmes include: - The header files of these scripts are in CrOS copyright format, but the the cros lint hook is configured to AoSP copyright format. - The test* functions in unittests are not compatible with CamelCase format. BUG=chromium:796338 TEST=unittests pass TEST=start_devserver TEST=cros flash TEST=scripts/paycheck.py Change-Id: I7eed4d1625eb7c510c7949fada120de5a6a26c7b Reviewed-on: https://chromium-review.googlesource.com/834875 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org> Reviewed-by: Sen Jiang <senj@chromium.org>
2018-01-04Fix pylint issue in payload.pySen Jiang
************* Module update_payload.payload W: 12, 0: Relative import 'applier', should be 'update_payload.applier' (relative-import) W: 13, 0: Relative import 'block_tracer', should be 'update_payload.block_tracer' (relative-import) W: 14, 0: Relative import 'checker', should be 'update_payload.checker' (relative-import) W: 15, 0: Relative import 'common', should be 'update_payload.common' (relative-import) W: 16, 0: Relative import 'error', should be 'update_payload.error' (relative-import) W: 17, 0: Relative import 'update_metadata_pb2', should be 'update_payload.update_metadata_pb2' (relative-import) C:222, 4: Missing function docstring (missing-docstring) E:261,33: Instance of 'DeltaArchiveManifest' has no 'partitions' member (no-member) Test: repo upload Change-Id: Ib3e99c6a3a5ea4575b442e672a871adc7b985289 (cherry picked from commit c2527f492e7a1389b2a3f248bd2b22d6cf5d9eb8) Reviewed-on: https://chromium-review.googlesource.com/834874 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Ben Chan <benchan@chromium.org> Reviewed-by: Sen Jiang <senj@chromium.org>
2017-12-01update_payload: Add puffin binary path to paycheck.pyAmin Hassani
We forgot to add the path to the puffin binary to paycheck.py and payload.py and this caused the canary builds fail for minor version 4. This patch fixes it. BUG=chromium:791079 TEST=tryjob Change-Id: I194ab9d9d0b93fc9d456b994f73424d7dedbf181 Reviewed-on: https://chromium-review.googlesource.com/803656 Commit-Ready: Amin Hassani <ahassani@chromium.org> Tested-by: Amin Hassani <ahassani@chromium.org> Reviewed-by: Andrew de los Reyes <adlr@chromium.org>
2017-09-30update_device.py: support OTA zip if use omaha too.Sen Jiang
Pass the offset and size to UpdateHandler. Also added support for payload offset in update_payload.Payload. Test: applied a local OTA Change-Id: Ib116ef2c23a11e298118f203814c4ea8dd1629af
2017-09-30Fix pylint issue in payload.pySen Jiang
************* Module update_payload.payload W: 12, 0: Relative import 'applier', should be 'update_payload.applier' (relative-import) W: 13, 0: Relative import 'block_tracer', should be 'update_payload.block_tracer' (relative-import) W: 14, 0: Relative import 'checker', should be 'update_payload.checker' (relative-import) W: 15, 0: Relative import 'common', should be 'update_payload.common' (relative-import) W: 16, 0: Relative import 'error', should be 'update_payload.error' (relative-import) W: 17, 0: Relative import 'update_metadata_pb2', should be 'update_payload.update_metadata_pb2' (relative-import) C:222, 4: Missing function docstring (missing-docstring) E:261,33: Instance of 'DeltaArchiveManifest' has no 'partitions' member (no-member) Test: repo upload Change-Id: Ib3e99c6a3a5ea4575b442e672a871adc7b985289
2015-11-17update_payload: Fix IsDelta to support payload v2.Sen Jiang
Added check for the old_partition_info field which is used by major version 2 payload. Other changes are needed in order to pass presubmit lint check. BUG=None TEST=Run devserver with a v2 delta payload. Change-Id: I3d7057fde44bd24d768177d5ed5c36bc2d4ca94b Reviewed-on: https://chromium-review.googlesource.com/312952 Commit-Ready: Sen Jiang <senj@chromium.org> Tested-by: Sen Jiang <senj@chromium.org> Reviewed-by: Alex Deymo <deymo@chromium.org> Reviewed-by: Gilad Arnold <garnold@chromium.org>
2015-10-15Parse Payload v2 header.Alex Deymo
The update payload v2 contains an extra field in the header with the size of the metadata signatures and the metadata signatures stored right after the metadata. This patch parses the new payload format. BUG=b:22024447 TEST=cros payload show payload-v2.bin; served a payload v2 with devserver.py Change-Id: I8ce85af1df505f82f62a9d1cd57910cee6921f84 Reviewed-on: https://chromium-review.googlesource.com/306090 Commit-Ready: Alex Deymo <deymo@chromium.org> Tested-by: Alex Deymo <deymo@chromium.org> Reviewed-by: Gilad Arnold <garnold@chromium.org>
2013-08-26paycheck: allow to pass an explicit path to bspatchGilad Arnold
The bspatch binary is used when applying update payloads. By default, we were using whatever bspatch that was found via path expansion in os.execvp, however there are cases where we want to be specific about where the bspatch binary is that we need to be using (such as during paygen runs). BUG=chromium:277072 TEST=Non-default bspatch binary used Change-Id: I85ffd28aeb26938cbf5ea428fa97d29af0353a7d Reviewed-on: https://gerrit.chromium.org/gerrit/66736 Tested-by: Gilad Arnold <garnold@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2013-05-24paycheck: truncate partitions resulting from applying a payloadGilad Arnold
Since the correctness of the result only encompasses the filesystem (or otherwise "meaningful data") on the target partition, it is desirable to actually get rid of whatever is past that point. There are different reasons for the presence of such extra space in delta updates, including remnants from a source partition that served as baseline for a delta update, or scratch space used by MOVE operations for breaking cycles. We make truncation the default behavior, although it can be suppressed by passing the right flag (truncate_to_expected_size=False). Note that this change is necessary for comparing the results of applying a payload to the partitions as they are extracted from a target image, which is to be attempted during payload generation. This also fixes tiny gpylint complaints. BUG=chromium:241283 TEST=Emitted partition files truncated as expected Change-Id: Ibb71e4f2305ec41224afdc503168ae02c312f6fe Reviewed-on: https://gerrit.chromium.org/gerrit/56532 Tested-by: Gilad Arnold <garnold@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org> Reviewed-by: Gilad Arnold <garnold@chromium.org>
2013-05-15Update paycheck to understand the updated update_metadata fields.Don Garrett
When checking a payload, always start by printing a description of the payload being checked. Recompiled (with protoc) the updated update_metadata.proto from the update_engine. BUG=chromium:226310 TEST=Manual Run CQ-DEPEND=CL:47347 Change-Id: Ib60c6e6978d30444db7b65ef6d09540c9ffacdb8 Reviewed-on: https://gerrit.chromium.org/gerrit/50899 Commit-Queue: Don Garrett <dgarrett@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Tested-by: Don Garrett <dgarrett@chromium.org>
2013-05-13paycheck: support for in-place BSDIFF operationsGilad Arnold
When applying BSDIFF operations in payloads, we used to extract the block sequences from the old partition and serialize it into a temporary file, and vice versa for the new partition. This worked but did not allow us to test bspatch as it is actually being used by the update engine. This CLs allows paycheck to invoke bspatch so that it reads/writes block extents directly from/to the partition file, in the same way it's being used by the update engine. Since performance is the same, this is the new default behavior; users can opt for the aforementioned old behavior using a command-line flag (-x or --extract-bsdiff). BUG=chromium:229705 TEST=bspatch invoked differently depending on the -x flag TEST=Passes unit/integration tests Change-Id: I8821754e1163b357617ece6befa42d1c2e575930 Reviewed-on: https://gerrit.chromium.org/gerrit/50486 Tested-by: Gilad Arnold <garnold@chromium.org> Reviewed-by: Darin Petkov <petkov@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2013-05-13paycheck: src/dst -> old/new in applier codeGilad Arnold
The old/new terminology is consistent with that used in update payload manifest, as well as other parts of the update_payload library (e.g. checker). BUG=None TEST=Unit/integration test passes Change-Id: I91244ae8adf073b958e8cd7e7670341f056c848e Reviewed-on: https://gerrit.chromium.org/gerrit/50130 Tested-by: Gilad Arnold <garnold@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2013-05-07paycheck: enforce physical partition size correctlyGilad Arnold
During payload checking, payload has wrongly interpreted the size reported in the update payload to be the physical partition size, whereas this is in fact the size of the filesystem portion only (a misnomer). This sometimes caused it to emit errors on out-of-bounds operations, which are otherwise harmless in real-world scenarios. This CL makes a clear distinction between the two, with the following semantics: - The payload's embedded filesystem size must by <= the physical partition sizes. - Reading/writing from/to the new partition must be within the physical partition size boundaries, and not the filesystem ones. - Reading from the old partition is only allowed from filesystem boundaries; this is unchanged from current behavior and appears to be consistent with how we perform delta updates. - Old/new SHA256 verification during payload application is now limited to the allotted filesystem portion only (and not the full partition size). This is consistent with the update engine's semantics. - Other than that, this change currently has no further effect on payload application, which remains more permissive wrt to partition sizes. This also means that the sizes of partitions resulting from a payload application will not necessarily abide by the predetermined physical partition sizes. This is in line with the prevailing division of responsibilities between payload checking (strict) and application (relaxed). BUG=chromium:221847 TEST=Payload checking respects partition size override TEST=Unit tests pass TEST=Integration tests pass Change-Id: I0dbc88d538c0cc53b7551f4dfa8f543bcf480cd5 Reviewed-on: https://gerrit.chromium.org/gerrit/50103 Reviewed-by: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org> Commit-Queue: David James <davidjames@chromium.org>
2013-05-02paycheck: allow to disable specific checksGilad Arnold
This became necessary as the delta generator appears to generate payloads that fail certain checks (e.g. during update_engine unit testing). BUG=None TEST=Disabled checks not being triggered Change-Id: I4491e0cb32ef44f85e11ffb0402b40d1371525ae Reviewed-on: https://gerrit.chromium.org/gerrit/49676 Tested-by: Gilad Arnold <garnold@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2013-04-05paycheck: unit tests + fixes to checker moduleGilad Arnold
This adds missing unit tests for the checker module, bundled with fixes to some bugs that surfaced due to unit tests. This includes: * A fake extent (signified by start_block == UINT64_MAX) that accompanies a signature data blob bears different requirements than previously implemented. Specifically, the extent sequence must have exactly one extent; and the number of blocks is not necessarily one, rather it is the correct number that corresponds to the actual length of the signature blob. * REPLACE/REPLACE_BZ operations must contain data. * MOVE operation validation must ensure that all of the actual message extents are being used. * BSDIFF operation must contain data (the diff). * Signature pseudo-operation should be a REPLACE. BUG=chromium-os:34911,chromium-os:33607,chromium-os:7597 TEST=Passes unittests (upcoming); works with actual payloads. Change-Id: I4d839d1d4da1fbb4a493b208958a139368e2c8ca Reviewed-on: https://gerrit.chromium.org/gerrit/45429 Tested-by: Gilad Arnold <garnold@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
2013-03-08Update payload library + command-line toolGilad Arnold
An initial implementation of a Python module for parsing, checking and applying a Chrome OS update payload. Comes with a command-line tool (paycheck.py) for applying such operations on payload files, and a test script (test_paycheck.sh) for ensuring that the library and tool are working correctly. Since update_payload is introduced as a package, we're moving some previously merged utilities into the package's directory. (Unit testing for this code will be uploaded on a separate CL; see chromium-os:39663) BUG=chromium-os:34911,chromium-os:33607,chromium-os:7597 TEST=test_paycheck.sh successful on MP-signed payloads CQ-DEPEND=I5746a1d80e822a575f0d96f94d0b4e765fc64507 Change-Id: I77123a1fffbb2059c239b7145c6922968fdffb6a Reviewed-on: https://gerrit.chromium.org/gerrit/43041 Reviewed-by: Gilad Arnold <garnold@chromium.org> Tested-by: Gilad Arnold <garnold@chromium.org> Reviewed-by: Chris Sosa <sosa@chromium.org> Reviewed-by: Jay Srinivasan <jaysri@chromium.org> Reviewed-by: Don Garrett <dgarrett@chromium.org> Commit-Queue: Gilad Arnold <garnold@chromium.org>
generated by cgit v1.2.3 (git 2.25.1) at 2025-10-03 17:21:38 +0000