Skip to content

Comments

fix: fix search in memory with AnyEmbedding#1696

Merged
JoanFM merged 6 commits intomainfrom
anytensor-in-memory-search-fix
Jul 12, 2023
Merged

fix: fix search in memory with AnyEmbedding#1696
JoanFM merged 6 commits intomainfrom
anytensor-in-memory-search-fix

Conversation

@JoanFM
Copy link
Member

@JoanFM JoanFM commented Jul 10, 2023 •
edited
Loading

Allows to do easy search in Memory when AnyEmbedding is used as with TextDoc or ImageDoc:

index = InMemoryExactNNIndex[TextDoc]()

docs = DocList[TextDoc](
    [TextDoc(text='hey', embedding=np.random.rand(128)) for _ in range(200)]
)
index.index(docs)
res = index.find_batched(docs[0:10], search_field='embedding', limit=5)

@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from e484ff0 to f96171f Compare July 10, 2023 17:01
fix: allow find in in memory index with AnyEmbedding
bc0e40d 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from f96171f to bc0e40d Compare July 10, 2023 17:11
jupyterjazz
jupyterjazz reviewed Jul 10, 2023
View reviewed changes
Copy link
Contributor

@jupyterjazz jupyterjazz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why was AnyEmbedding failing?

@JoanFM
Copy link
Member Author

JoanFM commented Jul 10, 2023

why was AnyEmbedding failing?

It does not have a computational backend attached, so it cannot stack or do anything on tensors

@JoanFM JoanFM marked this pull request as draft July 10, 2023 22:22
samsja
samsja reviewed Jul 11, 2023
View reviewed changes
Copy link
Member

@samsja samsja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be better to change

_extract_embedding directly to look at the type to the AnyTensor

@JoanFM
Copy link
Member Author

JoanFM commented Jul 11, 2023

I think it would be better to change

_extract_embedding directly to look at the type to the AnyTensor

and what about the compBackend, is it okey If I return the NumpyBackend for AnyTensor as default?

test: add tests for torch and tf
e7315e8 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from d36a0b5 to e7315e8 Compare July 11, 2023 07:24
@github-actions github-actions bot added size/m and removed size/s labels Jul 11, 2023
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch 2 times, most recently from a41e3f7 to 4e013b4 Compare July 11, 2023 10:26
fix: try to pass all mypy checks
2babe51 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from 4e013b4 to 2babe51 Compare July 11, 2023 10:51
@JoanFM JoanFM requested review from jupyterjazz and samsja July 11, 2023 10:56
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from 4d8a20a to 57bf37b Compare July 11, 2023 13:47
test: undo changes in docker compose
bab8099 
Signed-off-by: Joan Fontanals Martinez <joan.martinez@jina.ai>
@JoanFM JoanFM force-pushed the anytensor-in-memory-search-fix branch from 57bf37b to bab8099 Compare July 11, 2023 14:30
@JoanFM JoanFM marked this pull request as ready for review July 11, 2023 14:46
@github-actions
Copy link

github-actions bot commented Jul 11, 2023

📝 Docs are deployed on https://ft-anytensor-in-memory-search-fix--jina-docs.netlify.app 🎉

@JoanFM JoanFM merged commit 94a479e into main Jul 12, 2023
@JoanFM JoanFM deleted the anytensor-in-memory-search-fix branch July 12, 2023 07:11
@JoanFM JoanFM linked an issue Jul 13, 2023 that may be closed by this pull request
TextDoc not easily indexable with InMemory and HNSWLib #1695
Closed
6 tasks
@JoanFM JoanFM mentioned this pull request Jul 14, 2023
Merged
8 tasks
@jupyterjazz jupyterjazz mentioned this pull request Jul 18, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@samsja samsja samsja approved these changes

@JohannesMessner JohannesMessner Awaiting requested review from JohannesMessner

@jupyterjazz jupyterjazz Awaiting requested review from jupyterjazz

Assignees

No one assigned

Labels

area/core area/testing area/typing component/array size/m

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

TextDoc not easily indexable with InMemory and HNSWLib

3 participants

@JoanFM @jupyterjazz @samsja