Semantic Kernel を利用して PosgreSQL を利用してベクトル検索をしてみる

今回は掲題通り、Semanic Kernel を利用して PosgreSQL をベクトルストアとして利用してみたいと思います。前回のポストでは Azure Cognitive Search をベクトルストアとして利用しようと試みましたが、connecter の成熟度が現時点（2023年7月7日現在）はイマイチなこともあり、こちらの活用は難しい状況でした。
normalian.hatenablog.com

こちらに対して、現時点で Azure 上でベクトルストアとして利用できる組み込みサービスは現時点では以下となる認識です。

• Azure Cosmos DB for MongoDB cluster
  • public preview
  • EastUS, West Europe, Australia East
• Azure Cognitive Search
  • private preview
  • 7月9日追記：public previewになった模様 Vector search - Azure Cognitive Search | Microsoft Learn
  • Japan East/Japan West で利用可能
• Azure Redis
  • RedisSearch は GA https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-redis-modules
  • Japan region では利用不可 https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/?products=redis-cache&regions=all`
• Azure Database for PostgreSQL - フレキシブル サーバ
  • GA
  • Japan region で利用可 https://learn.microsoft.com/ja-jp/azure/postgresql/flexible-server/how-to-use-pgvector

日本で利用することを考える場合、Azure Database for PostgreSQL - フレキシブル サーバが良さそうな感じがしています。そこで今回は Semantic Kernel を利用して Azure Database for PostgreSQLをベクトルデータストアとして活用します。

daisami@mysurfacebook1:~$ psql -h your-server-name.postgres.database.azure.com -p 5432 -U myadminuser sk_demo
Password for user myadminuser:
psql (12.15 (Ubuntu 12.15-0ubuntu0.20.04.1), server 14.7)
WARNING: psql major version 12, server major version 14.
         Some psql features might not work.
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

sk_demo=> CREATE EXTENSION IF NOT EXISTS "vector";
NOTICE:  extension "vector" already exists, skipping
CREATE EXTENSION
sk_demo=> CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
NOTICE:  extension "uuid-ossp" already exists, skipping
CREATE EXTENSION
sk_demo=>

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.Memory.Postgres;
using Microsoft.SemanticKernel.Memory;
using Npgsql;
using Pgvector.Npgsql;

namespace MyOpenAITest
{
    classProgram
    { 
        publicstaticasync Task Main(string[] args) {
            // Azure OpenAI parametersconststring deploymentName_chat ="text-davinci-003";
            conststring endpoint ="https://your-endpoint.openai.azure.com/";
            conststring openAiKey ="open-ai-key";
            conststring deploymentName_embedding ="text-embedding-ada-002";

            // posgresql connection stringconststring connectionString ="Host=your-server-name.postgres.database.azure.com;Port=5432;Database=sk_demo;User Id=your-username;Password=your-password";

            Console.WriteLine("== Start Applicaiton ==");

            NpgsqlDataSourceBuilder dataSourceBuilder =new NpgsqlDataSourceBuilder(connectionString);
            dataSourceBuilder.UseVector();
            NpgsqlDataSource dataSource = dataSourceBuilder.Build();
            PostgresMemoryStore memoryStore =new PostgresMemoryStore(dataSource, vectorSize:1536/*, schema: "public" */);

            IKernel kernel = Kernel.Builder
                // .WithLogger(ConsoleLogger.Log)
                .WithAzureTextCompletionService(deploymentName_chat, endpoint, openAiKey)
                .WithAzureTextEmbeddingGenerationService(deploymentName_embedding, endpoint, openAiKey)
                .WithMemoryStorage(memoryStore)
                .WithPostgresMemoryStore(dataSource, vectorSize:1536, schema:"public")
                .Build();

            conststring memoryCollectionName ="SKGitHub";

            var githubFiles =new Dictionary<string, string>()
            {
                ["https://github.com/microsoft/semantic-kernel/blob/main/README.md"]
                    ="README: Installation, getting started, and how to contribute",
                ["https://github.com/microsoft/semantic-kernel/blob/main/samples/notebooks/dotnet/02-running-prompts-from-file.ipynb"]
                    ="Jupyter notebook describing how to pass prompts from a file to a semantic skill or function",
                ["https://github.com/microsoft/semantic-kernel/blob/main/samples/notebooks/dotnet/00-getting-started.ipynb"]
                    ="Jupyter notebook describing how to get started with the Semantic Kernel",
                ["https://github.com/microsoft/semantic-kernel/tree/main/samples/skills/ChatSkill/ChatGPT"]
                    ="Sample demonstrating how to create a chat skill interfacing with ChatGPT",
                ["https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/SemanticKernel/Memory/Volatile/VolatileMemoryStore.cs"]
                    ="C# class that defines a volatile embedding store",
                ["https://github.com/microsoft/semantic-kernel/tree/main/samples/dotnet/KernelHttpServer/README.md"]
                    ="README: How to set up a Semantic Kernel Service API using Azure Function Runtime v4",
                ["https://github.com/microsoft/semantic-kernel/tree/main/samples/apps/chat-summary-webapp-react/README.md"]
                    ="README: README associated with a sample starter react-based chat summary webapp",
            };

            Console.WriteLine("Adding some GitHub file URLs and their descriptions to a volatile Semantic Memory.");
            int i =0;
            foreach (var entry in githubFiles)
            {
                await kernel.Memory.SaveReferenceAsync(
                    collection: memoryCollectionName,
                    description: entry.Value,
                    text: entry.Value,
                    externalId: entry.Key,
                    externalSourceName:"GitHub"
                );
                Console.WriteLine($"  URL {++i} saved");
            }

            string ask ="I love Jupyter notebooks, how should I get started?";
            Console.WriteLine("===========================\n"+"Query: "+ ask +"\n");

            var memories = kernel.Memory.SearchAsync(memoryCollectionName, ask, limit:5, minRelevanceScore:0.77);

            int j =0;
            awaitforeach (MemoryQueryResult memory in memories)
            {
                Console.WriteLine($"Result {++j}:");
                Console.WriteLine("  URL:     : "+ memory.Metadata.Id);
                Console.WriteLine("  Title    : "+ memory.Metadata.Description);
                Console.WriteLine("  Relevance: "+ memory.Relevance);
                Console.WriteLine();
            }

            Console.WriteLine("== End Application ==");
        }
    }
}