Skip to content

Fix UTF-8 codepoint counting in cbor_string_set_handle and parsing - #286

Merged
PJK merged 8 commits into
masterfrom
codepoint-fixes
Jun 3, 2023
Merged

PJK merged 8 commits into
masterfrom
codepoint-fixes

Conversation

@PJK

@PJK PJK commented Jun 3, 2023

Copy link
Copy Markdown
Owner
  • Fix value not being set by cbor_string_set_handle
  • Do not fail parsing on a well-formed (https://www.rfc-editor.org/rfc/rfc8949#section-3.1-2.8 says "A string containing an invalid UTF-8 sequence is well-formed but invalid") to be consistent in the parsing and manipulation APIs
  • Document the new behavior

@codecov

codecov Bot commented Jun 3, 2023 •

Copy link
Copy Markdown

Codecov Report

Merging #286 (d9f60b3) into master (9ac3066) will decrease coverage by 0.13%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #286      +/-   ##
==========================================
- Coverage   99.39%   99.27%   -0.13%     
==========================================
  Files          20       20              
  Lines        1665     1664       -1     
==========================================
- Hits         1655     1652       -3     
- Misses         10       12       +2     
Impacted Files Coverage Δ
src/cbor/internal/builder_callbacks.c 97.70% <ø> (-0.06%) ⬇️
src/cbor/internal/unicode.c 100.00% <100.00%> (ø)
src/cbor/strings.c 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@PJK
PJK merged commit 5fe9656 into master Jun 3, 2023
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Feb 4, 2024
0.11.0 (2024-02-04)
---------------------
- [Updated documentation to refer to RFC 8949](PJK/libcbor#269)
- Improvements to `cbor_describe`
  - [Bytestring data will now be printed as well](PJK/libcbor#281) by  [akallabeth](https://github.com/akallabeth)
  - [Formatting consistency and clarity improvements](PJK/libcbor#285)
- [Fix `cbor_string_set_handle` not setting the codepoint count](PJK/libcbor#286)
- BREAKING: [`cbor_load` will no longer fail on input strings that are well-formed but not valid UTF-8](PJK/libcbor#286)
  - If you were relying on the validation, please check the result using `cbor_string_codepoint_count` instead
- BREAKING: [All decoders like `cbor_load` and `cbor_stream_decode` will accept all well-formed tag values](PJK/libcbor#308) (bug discovered by [dskern-github](https://github.com/dskern-github))
  - Previously, decoding of certain values would fail with `CBOR_ERR_MALFORMATED` or `CBOR_DECODER_ERROR`
  - This also makes decoding symmetrical with serialization, which already accepts all values
msk pushed a commit to msk/pkgsrc that referenced this pull request May 11, 2026
0.11.0 (2024-02-04)
---------------------
- [Updated documentation to refer to RFC 8949](PJK/libcbor#269)
- Improvements to `cbor_describe`
  - [Bytestring data will now be printed as well](PJK/libcbor#281) by  [akallabeth](https://github.com/akallabeth)
  - [Formatting consistency and clarity improvements](PJK/libcbor#285)
- [Fix `cbor_string_set_handle` not setting the codepoint count](PJK/libcbor#286)
- BREAKING: [`cbor_load` will no longer fail on input strings that are well-formed but not valid UTF-8](PJK/libcbor#286)
  - If you were relying on the validation, please check the result using `cbor_string_codepoint_count` instead
- BREAKING: [All decoders like `cbor_load` and `cbor_stream_decode` will accept all well-formed tag values](PJK/libcbor#308) (bug discovered by [dskern-github](https://github.com/dskern-github))
  - Previously, decoding of certain values would fail with `CBOR_ERR_MALFORMATED` or `CBOR_DECODER_ERROR`
  - This also makes decoding symmetrical with serialization, which already accepts all values
jperkin pushed a commit to TritonDataCenter/pkgsrc that referenced this pull request May 14, 2026
0.11.0 (2024-02-04)
---------------------
- [Updated documentation to refer to RFC 8949](PJK/libcbor#269)
- Improvements to `cbor_describe`
  - [Bytestring data will now be printed as well](PJK/libcbor#281) by  [akallabeth](https://github.com/akallabeth)
  - [Formatting consistency and clarity improvements](PJK/libcbor#285)
- [Fix `cbor_string_set_handle` not setting the codepoint count](PJK/libcbor#286)
- BREAKING: [`cbor_load` will no longer fail on input strings that are well-formed but not valid UTF-8](PJK/libcbor#286)
  - If you were relying on the validation, please check the result using `cbor_string_codepoint_count` instead
- BREAKING: [All decoders like `cbor_load` and `cbor_stream_decode` will accept all well-formed tag values](PJK/libcbor#308) (bug discovered by [dskern-github](https://github.com/dskern-github))
  - Previously, decoding of certain values would fail with `CBOR_ERR_MALFORMATED` or `CBOR_DECODER_ERROR`
  - This also makes decoding symmetrical with serialization, which already accepts all values
PJK added a commit that referenced this pull request Sep 16, 2026
The strings documentation still claimed that invalid UTF-8 is rejected
by the parser, which has not been true since 0.11.0 (#286). Replace it
with an accurate description of cbor_string_codepoint_count-based
checking and add a single section in decoding.rst stating the policy:
decoders check RFC 8949 Section 3 well-formedness only, and Section 5.3
validity (UTF-8 encoding, duplicate map keys, tag content types) is left
to the application. Cross-reference it from the strings, maps, and tags
pages.

Prompted by #440.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
tsutsui pushed a commit to tsutsui/pkgsrc that referenced this pull request Oct 17, 2026
0.11.0 (2024-02-04)
---------------------
- [Updated documentation to refer to RFC 8949](PJK/libcbor#269)
- Improvements to `cbor_describe`
  - [Bytestring data will now be printed as well](PJK/libcbor#281) by  [akallabeth](https://github.com/akallabeth)
  - [Formatting consistency and clarity improvements](PJK/libcbor#285)
- [Fix `cbor_string_set_handle` not setting the codepoint count](PJK/libcbor#286)
- BREAKING: [`cbor_load` will no longer fail on input strings that are well-formed but not valid UTF-8](PJK/libcbor#286)
  - If you were relying on the validation, please check the result using `cbor_string_codepoint_count` instead
- BREAKING: [All decoders like `cbor_load` and `cbor_stream_decode` will accept all well-formed tag values](PJK/libcbor#308) (bug discovered by [dskern-github](https://github.com/dskern-github))
  - Previously, decoding of certain values would fail with `CBOR_ERR_MALFORMATED` or `CBOR_DECODER_ERROR`
  - This also makes decoding symmetrical with serialization, which already accepts all values
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

⌂ 换网址