You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
getAll() with both key and value (or index key and primary key) #206
As a cursor replacement, getAll()/getAllKeys() has the issue that you need to pick - you can't get both the key and the value. For object stores this isn't so bad if inline keys are used, but still awkward. For indexes you can't even get the primary key.
Here's an example where getAll() could help but needs index key and primary key
Having getAllEntries() on an Index produce [[primaryKey, value], ...] is closest to how getAllXXX() work on Indexes today, but there are use cases for the other permutations as well.
I put a PR up for a straightforward version of getAllEntries() but see above re: index use cases. No consensus on this yet, I was mostly playing with refactoring the methods.
Staring at this a bit more, I'd be tempted to have indexes produce:
[[primaryKey, value, indexKey], ... ]
i.e. if you only pay attention to the first two elements, this looks just like getAllEntries() and can be used with Map constructor. If you need the index key it's there.
By default IndexedDb seems to store by ascending order, so I guess prev would be descending or reverse order. This should be fairly straightforward to do with leveldb, I have the changes locally (albeit against an earlier version of chromium) if this change has traction, then am willing to go ahead and open a PR for chromium to add it in, need to see what should be done for FF/Webkit browsers though.
On a related note, this is something that would be extremely useful with messaging apps, by default we store messages in ascending time order (next cursor) but we often need to show them in descending time order in the UI (latest message first or by using the prev cursor).
For the index case, in my opinion, option 2 (indexKey, primaryKey) would be best, as this option would allow us to obtain the index keys for a range without the overhead of fetching full values.
I am not sure if this is covered by "key" value for the "query" option? Would be great if this could be clarified!
+1 about "record" being a better term for getAll...() when applied to regular object stores. But for indexes, IIUC, "record" => [indexKey, primaryKey]. If we're agreed on returning [primaryKey, value, indexKey] from IDBIndex.getAll...(), then the contract of the new API isn't exactly about returning the records as-is from the underlying (regular/index) object stores, but about returning a collection of attributes that we hope to be useful for both types of stores, and it just so happens to be the same as the record for regular object stores. From that perspective, it maybe better to use a new term, and "entry" could serve that purpose.
On naming: IDBIndex's getAllKeys() already uses "key" to refer to the object store record's key (i.e. primary key), not the index's key. Likewise getAll() returns referenced values i.e. the value half of an object store record. So the established getAllXXX terminology is relative to the referenced object store. (Well technically getAll() should probably be called getAllValues() and then we could just call the new thing getAll(), but anyway...)
On form:
If we're agreed on returning [primaryKey, value, indexKey]
I think we are aligned on that function but perhaps not the form. It seems to me a bit unclean thing to return an array of a particular length where you just have to know which element is which thing, as opposed to an object such as
About the form { key: value }, I don't think this will work for all types of IndexedDB keys. For example, binary keys and array keys may cause issues.
For consistency, should we add a record property to IDBCursorWithValue? This property could return the same type as the equivalent value in the getAllRecords() result array?
Here are a few options we could consider for the form of the results:
Continuing with getAllEntries(), update IDBIndex::getAllEntries() to return an array of nested entries:
Consistency: getAllEntries() returns an array of entries for both IDBObjectStore and IDBIndex.
Compatible with existing ECMAScript entries. Like @inexorabletash pointed out above, developers may directly use the results to construct a Map or Object.
Disadvantages:
Duplicate keys exist in the entries array for indexes with non-unique keys.
Unusual ergonomics. Like @evanstade pointed out above; to access the value of the second record, developers would use results[1][1][1], which is not very intuitive.
Like an IDBCursor, for IDBObjectStore, key and primaryKey return the same value, which is the record's primary key. For IDBIndex, key returns the index key while primaryKey returns the record's primary key.
Continuing the example above, developers would use the following to access the value of the second record: results[1].value.
Like @inexorabletash pointed out above, developers have a lot of flexibility when using Array.map() with destructuring. For example, developers can construct a Map:
// Transform the results from `getAllRecords()` into an array of entries: // [ [ primaryKey1, { 'value': value1, 'indexKey': indexKey1 }], [ primaryKey2, { 'value': value2, ... ], ... ]constmap=newMap(get_all_records_results.map(({primaryKey, value, key})=>[primaryKey,{ value,'indexKey': key}]));// Use the map to access record values and index keys.constvalue=map.get(primaryKey).value;constindexKey=map.get(primaryKey).indexKey;
I plan to update the explainer and prototype to use this format, but feedback is very welcome! I'm happy to continue to iterate.
For consistency, should we add a record property to IDBCursorWithValue? This property could return the same type as the equivalent value in the getAllRecords() result array?
Isn't the IDBCursorWithValue already more or less the record (or a superset of the record)? We could maybe add an asRecord() convenience method, not sure how much demand there is though.
Is there something that distinguishes an IDB "record" from the "entries" in Object.entries and Map.entries? If there's not, I suggest using the name the rest of the platform has already used.
while "record" is a spec term, I don't think it has made it into the API yet so there isn't a consistency argument.
My main argument here is that the spec and MDN (e.g.) both consistently use the term "record". Other spec terms tend to map directly into the API symbol names. To do otherwise would make it harder to reference the spec or MDN as documentation. But you are correct that if only looking at API symbol names, "record" hasn't been used yet.
Is there something that distinguishes an IDB "record" from the "entries" in Object.entries and Map.entries?
Here are a few:
Conceptually, records are immutable. You can only read them, delete them, or replace them.
Concretely
IDB records cannot be accessed synchronously
so far we're planning to use a different format for the data returned by getAllRecords as compared to Map.entries or Object.entries.
An IDB record is still conceptually a key/value pair, but "record" I think more strongly evokes exactly what it is, i.e. a "row in a database", than does "entry". TBH, either term work, so it feels like a bit of a bike shed and it's probably not possible to make a wrong decision here, but what is the concrete advantage of using "entry"?
I see there are some tests in WPT for this, so it would be nice if this could get an update, and maybe some official spec text. (I assume this was the start of that)
Hi @stelar7, getAllRecords() is in a developer trial for Chromium based browsers. The Edge team has received positive feedback from developers. We've also received feedback that we should add direction to the existing methods: getAll() and getAllKey(), which I am discussing in another issue. Afterwards, I'd like to advance to the next stage for Chromium.
I also plan to draft a spec change soon. I can CC you on the spec change. I tried adding you to my prior spec changes, but GitHub wouldn't let me so maybe you need to join the W3C to participate in pull requests.
Activity
inexorabletash commentedon Jun 5, 2019
A good name surfaced over in WICG/kv-storage#63 -
getAllEntries()... which sort-of aligns with ECMAScript's array iterators:
values(),keys(), andentries()It would produce an array of two-element arrays.
brettz9 commentedon Jun 5, 2019
Which could nicely be fed into
Object.fromEntries, passed into aMap, etc.inexorabletash commentedon Jun 7, 2019
Object stores are easy:
[[primaryKey, value], ...]Indexes are less straightforward:
[[primaryKey, value], ...][[indexKey, primaryKey], ...][[indexKey, primaryKey, value], ...]Having getAllEntries() on an Index produce
[[primaryKey, value], ...]is closest to how getAllXXX() work on Indexes today, but there are use cases for the other permutations as well.Define getAllEntries() methods. Issue #206
inexorabletash commentedon Jun 7, 2019
I put a PR up for a straightforward version of
getAllEntries()but see above re: index use cases. No consensus on this yet, I was mostly playing with refactoring the methods.inexorabletash commentedon Sep 20, 2019
TPAC 2019 Web Apps breakout:
Follow-up: refine proposal for indexes
inexorabletash commentedon Sep 20, 2019
Staring at this a bit more, I'd be tempted to have indexes produce:
[[primaryKey, value, indexKey], ... ]i.e. if you only pay attention to the first two elements, this looks just like getAllEntries() and can be used with Map constructor. If you need the index key it's there.
inexorabletash commentedon Dec 16, 2019
Sketching this out a bit more:
Note that this still doesn't allow trivial paging of results i.e. with a "skip".
directionoption would be IDBCursorDirection limited to"prev"or"next"(throwing otherwise)."prev"are the results in forward or reverse order?masterkidan commentedon Aug 8, 2020
By default IndexedDb seems to store by ascending order, so I guess prev would be descending or reverse order. This should be fairly straightforward to do with leveldb, I have the changes locally (albeit against an earlier version of chromium) if this change has traction, then am willing to go ahead and open a PR for chromium to add it in, need to see what should be done for FF/Webkit browsers though.
On a related note, this is something that would be extremely useful with messaging apps, by default we store messages in ascending time order (next cursor) but we often need to show them in descending time order in the UI (latest message first or by using the prev cursor).
stefanhaustein commentedon Oct 14, 2021
For the index case, in my opinion, option 2 (indexKey, primaryKey) would be best, as this option would allow us to obtain the index keys for a range without the overhead of fetching full values.
I am not sure if this is covered by "key" value for the "query" option? Would be great if this could be clarified!
5 remaining items
abhishek-shanthkumar commentedon Oct 9, 2024
+1 about "record" being a better term for
getAll...()when applied to regular object stores. But for indexes, IIUC, "record" =>[indexKey, primaryKey]. If we're agreed on returning[primaryKey, value, indexKey]fromIDBIndex.getAll...(), then the contract of the new API isn't exactly about returning the records as-is from the underlying (regular/index) object stores, but about returning a collection of attributes that we hope to be useful for both types of stores, and it just so happens to be the same as the record for regular object stores. From that perspective, it maybe better to use a new term, and "entry" could serve that purpose.evanstade commentedon Oct 9, 2024
Fair points by @abhishek-shanthkumar
On naming: IDBIndex's
getAllKeys()already uses "key" to refer to the object store record's key (i.e. primary key), not the index's key. LikewisegetAll()returns referenced values i.e. the value half of an object store record. So the establishedgetAllXXXterminology is relative to the referenced object store. (Well technicallygetAll()should probably be calledgetAllValues()and then we could just call the new thinggetAll(), but anyway...)On form:
I think we are aligned on that function but perhaps not the form. It seems to me a bit unclean thing to return an array of a particular length where you just have to know which element is which thing, as opposed to an object such as
or
SteveBeckerMSFT commentedon Oct 22, 2024
A few comments:
About the form
{ key: value }, I don't think this will work for all types of IndexedDB keys. For example, binary keys and array keys may cause issues.For consistency, should we add a
recordproperty toIDBCursorWithValue? This property could return the same type as the equivalent value in thegetAllRecords()result array?Here are a few options we could consider for the form of the results:
getAllEntries(), updateIDBIndex::getAllEntries()to return an array of nested entries:[[ indexKey1, [ primaryKey1, value1]], [ indexKey2, [ primaryKey2, value2]], ... ]Advantages:
getAllEntries()returns an array of entries for bothIDBObjectStoreandIDBIndex.Disadvantages:
results[1][1][1], which is not very intuitive.Like an
IDBCursor, forIDBObjectStore,keyandprimaryKeyreturn the same value, which is the record's primary key. ForIDBIndex,keyreturns the index key whileprimaryKeyreturns the record's primary key.Continuing the example above, developers would use the following to access the value of the second record:
results[1].value.Like @inexorabletash pointed out above, developers have a lot of flexibility when using Array.map() with destructuring. For example, developers can construct a
Map:I plan to update the explainer and prototype to use this format, but feedback is very welcome! I'm happy to continue to iterate.
evanstade commentedon Oct 24, 2024
Conclusions above SGTM.
Isn't the IDBCursorWithValue already more or less the record (or a superset of the record)? We could maybe add an
asRecord()convenience method, not sure how much demand there is though.jyasskin commentedon Nov 4, 2024
Is there something that distinguishes an IDB "record" from the "entries" in
Object.entriesandMap.entries? If there's not, I suggest using the name the rest of the platform has already used.inexorabletash commentedon Nov 5, 2024
Also, while "record" is a spec term, I don't think it has made it into the API yet so there isn't a consistency argument.
evanstade commentedon Nov 5, 2024
My main argument here is that the spec and MDN (e.g.) both consistently use the term "record". Other spec terms tend to map directly into the API symbol names. To do otherwise would make it harder to reference the spec or MDN as documentation. But you are correct that if only looking at API symbol names, "record" hasn't been used yet.
Here are a few:
Map.entriesorObject.entries.An IDB record is still conceptually a key/value pair, but "record" I think more strongly evokes exactly what it is, i.e. a "row in a database", than does "entry". TBH, either term work, so it feels like a bit of a bike shed and it's probably not possible to make a wrong decision here, but what is the concrete advantage of using "entry"?
stelar7 commentedon May 14, 2025
Hi all!
I see there are some tests in WPT for this, so it would be nice if this could get an update, and maybe some official spec text. (I assume this was the start of that)
SteveBeckerMSFT commentedon May 15, 2025
Hi @stelar7, getAllRecords() is in a developer trial for Chromium based browsers. The Edge team has received positive feedback from developers. We've also received feedback that we should add direction to the existing methods: getAll() and getAllKey(), which I am discussing in another issue. Afterwards, I'd like to advance to the next stage for Chromium.
I also plan to draft a spec change soon. I can CC you on the spec change. I tried adding you to my prior spec changes, but GitHub wouldn't let me so maybe you need to join the W3C to participate in pull requests.
Thanks!
Steve
stelar7 commentedon May 15, 2025
I'm not that familiar with the W3C setup wrt. trials, but a CC would be great 👍
As far as I'm aware, Im part of the Web Apps WG, but maybe there is some other thing I need to be part of
Thanks!
inexorabletash commentedon May 15, 2025
Now that @stelar7 is in the WG they should show up as an option for reviewer on PRs.
Add getAllRecords() method and update getAll()/getAllKeys() to suppor…
Add getAllRecords() method and update getAll()/getAllKeys() to suppor…
danowar2 commentedon Jul 31, 2025
Thanks for the update