feat: support redis by jupyterjazz · Pull Request #1550 · docarray/docarray

jupyterjazz · 2023-05-17T14:24:31Z

Redis as a Document Index Backend

Simple usage:

class MyDoc(BaseDoc):
    text: str
    embedding: NdArray[10]

docs = [MyDoc(text=f'text {i}', embedding=np.random.rand(10)) for i in range(10)]
query = np.random.rand(10)
db = RedisDocumentIndex[MyDoc](host='localhost')
db.index(docs)
results = index.find(query, search_field='embedding', limit=10)

Supported functionality:

Find (vector search)
Filter (filtering on textual and numeric data using redis syntax)
Text Search (text search using various approaches, e.g. BM25..)
Get/Del
Hybrid Search (currently, only combining find and filter)
Subindex

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

Signed-off-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com>

github-actions · 2023-06-28T07:13:43Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-06-28T07:21:30Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-06-28T08:01:20Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-06-28T08:02:00Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-06-28T08:44:37Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-06-28T09:01:33Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

docarray/index/backends/redis.py

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

docarray/index/backends/redis.py

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-07-05T07:12:27Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-07-05T07:12:59Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

docarray/index/backends/redis.py

JohannesMessner

Some small comments, but looking good!

docarray/index/backends/redis.py

JohannesMessner · 2023-07-06T06:57:11Z

docarray/index/backends/redis.py

+
+        :return: Number of documents in the index.
+        """
+        num_docs = self._client.ft(self._index_name).info()['num_docs']


How does this work? Maybe I'm wrong, but it seems like this 'num_docs' key is not something that is built into redis. Do we manually keep track of this somehow?

It is, num_docs is included in the index info

but apparently there's a better way to get number of docs
return self._client.dbsize()

But I think we should stick with the old way because dbsize returns number of keys the client stores, combining all indices so it's inaccurate if you have more than one index

JohannesMessner · 2023-07-06T06:58:23Z

docarray/index/backends/redis.py

+
+        ids = [self._prefix + id for id in doc_ids]
+        docs = self._client.json().mget(ids, '$')
+        docs = [doc[0] for doc in docs if doc]


Suggested change

docs = [doc[0] for doc in docs if doc]

docs = (doc[0] for doc in docs if doc)

I think this avoids doing an iteration over the data

yes but the abstract class expects a list and not a generator, we can refactor that in a separate pr maybe

docarray/index/backends/redis.py

JohannesMessner · 2023-07-06T07:02:58Z

docarray/index/backends/redis.py

+    def __contains__(self, item: BaseDoc) -> bool:
+        """
+        Checks if a given document exists in the index.
+
+        :param item: The document to check.
+            It must be an instance of BaseDoc or its subclass.
+        :return: True if the document exists in the index, False otherwise.
+        """
+        if safe_issubclass(type(item), BaseDoc):
+            return self._doc_exists(item.id)
+        else:
+            raise TypeError(
+                f"item must be an instance of BaseDoc or its subclass, not '{type(item).__name__}'"
+            )


Maybe worthy of a separete PR, but I think we should move __contains__ to the abstract class and tell backends to implement _doc_exitsts().

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-07-06T11:02:16Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-07-06T11:05:53Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-07-06T11:06:03Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

docarray/index/backends/redis.py

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-07-09T11:56:47Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-07-09T11:57:15Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions · 2023-07-09T13:28:55Z

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

github-actions · 2023-07-09T13:32:02Z

📝 Docs are deployed on https://ft-feat-add-redis--jina-docs.netlify.app 🎉

feat: support redis

df47474

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions bot added size/m area/core area/setup labels May 17, 2023

jupyterjazz changed the title ~~[wip ]feat: support redis~~ feat: support redis May 17, 2023

github-actions bot added the area/testing label May 17, 2023

jupyterjazz marked this pull request as draft May 17, 2023 14:25

jupyterjazz and others added 5 commits June 12, 2023 16:11

chore: merge main

0920adc

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

fix: index creation

51558b7

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

feat: 1st draft, needs polishing

12da714

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

feat: query builder, tests

be7bed7

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

Merge branch 'main' into feat-add-redis

9365327

Signed-off-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com>

github-actions bot added size/xl and removed size/m labels Jun 28, 2023

chore: update poetry lock

cb25869

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

chore: run tests

341fa9a

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

github-actions bot added area/cicd area/housekeeping labels Jun 28, 2023

Merge branch 'main' into feat-add-redis

d37b28f

jupyterjazz added 2 commits June 28, 2023 10:43

fix: defaultdict for column config

abca1dd

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

chore: update branch

edc8a34

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

style: ignore mypy errors

c5abd80

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

JoanFM requested changes Jun 28, 2023

View reviewed changes

docarray/index/backends/redis.py Outdated Show resolved Hide resolved

refactor: put vectorfield args in column info

22434f1

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

JoanFM requested changes Jul 5, 2023

View reviewed changes

docarray/index/backends/redis.py Outdated Show resolved Hide resolved

docarray/index/backends/redis.py Show resolved Hide resolved

chore: commits not showing

848e95d

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

Merge branch 'main' into feat-add-redis

c00abe2

JoanFM requested changes Jul 5, 2023

View reviewed changes

docarray/index/backends/redis.py Outdated Show resolved Hide resolved

docarray/index/backends/redis.py Show resolved Hide resolved

docarray/index/backends/redis.py Show resolved Hide resolved

JohannesMessner reviewed Jul 6, 2023

View reviewed changes

jupyterjazz mentioned this pull request Jul 6, 2023

feat: doc_exists for each backend #1686

Closed

jupyterjazz requested a review from JoanFM July 6, 2023 10:49

feat: del and get batched

27dd29a

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

jupyterjazz requested a review from JohannesMessner July 6, 2023 11:03

jupyterjazz and others added 2 commits July 6, 2023 15:05

docs: update batchsize docstring

162c6a8

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

Merge branch 'main' into feat-add-redis

16c4323

JoanFM requested changes Jul 7, 2023

View reviewed changes

docarray/index/backends/redis.py Outdated Show resolved Hide resolved

docarray/index/backends/redis.py Show resolved Hide resolved

refactor: index name

56829d6

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

Merge branch 'main' into feat-add-redis

c414836

jupyterjazz requested a review from JoanFM July 9, 2023 12:02

jupyterjazz added 2 commits July 9, 2023 17:27

refactor: default index name following schema

7a5ed5e

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

chore: update branch

0fecb48

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

JoanFM approved these changes Jul 10, 2023

View reviewed changes

JoanFM merged commit 069aa3a into main Jul 10, 2023

JoanFM deleted the feat-add-redis branch July 10, 2023 12:52

jupyterjazz mentioned this pull request Jul 18, 2023

Release Notes v0.36.0 #1707

Closed

	docs = [doc[0] for doc in docs if doc]
	docs = (doc[0] for doc in docs if doc)

Comments

Conversation

jupyterjazz commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Redis as a Document Index Backend

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

github-actions bot commented Jun 28, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 5, 2023

Uh oh!

github-actions bot commented Jul 5, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JohannesMessner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

JohannesMessner Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

jupyterjazz Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

jupyterjazz Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

JohannesMessner Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

jupyterjazz Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JohannesMessner Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

jupyterjazz Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 6, 2023

Uh oh!

github-actions bot commented Jul 6, 2023

Uh oh!

github-actions bot commented Jul 6, 2023

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 9, 2023

Uh oh!

github-actions bot commented Jul 9, 2023

Uh oh!

github-actions bot commented Jul 9, 2023

Uh oh!

github-actions bot commented Jul 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

jupyterjazz commented May 17, 2023 •

edited

Loading