semantica-agi/semantica
Graph-Native Infrastructure for Context and Accountable AI Systems - semantica-agi/semantica
9.5k stars · 993 forks
README
Graph-Native Infrastructure for Context and Accountable AI Systems The Open Source Palantir for AI Agents
Ingest your enterprise data, extract what matters, build a Context Graph and knowledge graph (KG), and run graph analytics and causal reasoning over all of it, with full decision provenance baked in. Explainable, traceable, and trustworthy by design.
Decision Intelligence · Context Management · Deterministic Reasoning · Ontology Management · Knowledge Modeling · End-to-End Traceability Open Source · Self-Hostable · Auditable · Governed · Zero Vendor Lock-In Polyglot Graph Storage · RDF & LPG Support · W3C Standards · Interoperable Built for High-Stakes, Regulated Domains pip install semantica
Knowledge Explorer · Context Graphs · Reasoning Engine · Decision Intelligence · Ontology Hub ▶ Watch the full platform walkthrough
Most AI agents act without a trail. They store embeddings, not meaning: context that can't be explained, decisions that can't be audited. In lending, that gap is a compliance exposure, not an inconvenience: an underwriting agent's approval has to survive a regulator's "why" months later. Semantica sits underneath your LLM, vector store, and agent framework as a deterministic infrastructure layer: no LLM required for graph construction, reasoning, or provenance.
⚠️ System-level explainability, not foundation-model explainability. Semantica does not expose or reconstruct what happens inside the LLM — its internal reasoning or chain-of-thought stays opaque, as it does for any external system. Semantica explains what's outside the model: the context and data fed in, the decision produced, its provenance, relevant relationships, applied policies, and the full execution trail.
Who it's for:
AI/ML platform teams shipping agents that make consequential decisions and need structured, queryable context built from fragmented raw data, not just a vector index Data platform teams on Databricks or Snowflake who need to turn tables already sitting in Unity Catalog or a Snowflake warehouse into a governed, lineage-tracked knowledge graph, without exporting that data to a third-party SaaS first Compliance, risk, and audit teams who need a straight answer to "why did the AI do that?" in a format a regulator will actually accept Regulated enterprises (finance, healthcare, legal, government, defense) that can't ship a black box, and can't send their data to someone else's SaaS to get one Platform and infra engineers who want the KG, reasoning, and provenance stack self-hosted and swappable, not locked to one vendor's backend Data and knowledge engineers building a KG from messy, multi-source data: entities and relationships get extracted, conflicting or contradictory facts are flagged instead of silently overwritten, and duplicates are merged before they turn into noise
Quick Start · Architecture · What You Get · Why Semantica · Decision Intelligence · Context Graphs · Recipe: Audit Trail · Module Reference · Integrations · CLI · Performance · Install
What Semantica Gives You
Context Graphs: A structured, queryable graph of everything your agent knows, decides, and reasons about Decision Intelligence: Every decision is a first-class object: traceable, searchable by precedent, and causally linked AI Governance & Ontology: SHACL constraints, conflict detection, compliance rules, OWL generation, and SKOS vocabulary management with a visual editor Full Auditability: W3C PROV-O provenance on every fact, with audit trails exportable to JSON, CSV, or RDF Deterministic Reasoning: Forward chaining, Rete network, Datalog, and SPARQL with fully explainable paths, not black boxes Knowledge Pipeline: Multi-source ingestion, entity-aware chunking, NER/relation/event extraction, and knowledge graph construction, with semantic deduplication and provenance-preserving merges throughout Enterprise Data Platforms: Native connectors for Databricks (Unity Catalog + Delta Lake, PAT/OAuth M2M auth, catalog/schema/table/lineage introspection) and Snowflake (warehouse/database/schema, key-pair and OAuth auth), so tables already living in your lakehouse or warehouse become graph nodes with provenance, not another export/import hop Graph Analytics: Centrality, community detection, link prediction, and shortest-path queries over the graph you just built Polyglot Graph Storage: Native RDF (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J via SPARQL) and Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune via Cypher), plus vector stores, all swappable without touching your code Visualization: Explore any graph, ontology, or timeline in an interactive browser workbench Drop-in Integrations: Native Agno and CrewAI support, a full-featured MCP server, a comprehensive CLI, a REST API, and plugins across major editors
Why Semantica
Vector DB + RAG Plain LLM Memory Semantica
Recall method Embedding similarity Token window Graph traversal + semantic search
Decision history Not stored Not stored First-class queryable objects
Provenance None None W3C PROV-O, source-linked
Reasoning None Black box Forward chain, Rete, Datalog, SPARQL
Conflict detection Silent overwrite Silent overwrite Detected, flagged, resolved
Time travel No No Point-in-time graph snapshots
Compliance export None None PROV-O, SHACL, OWL, RDF
Policy enforcement None None Built-in rule engine + SHACL
Entity resolution No No Blocking + semantic deduplication
Multi-agent context Separate per agent Separate per agent Single shared intelligence layer
Semantica complements your existing stack rather than replacing it. Keep your LLM, vector store, and agent framework exactly as they are; Semantica adds the decision records, causal reasoning, provenance, ontology governance, conflict detection, and audit trails on top. The reasoning engines, KG construction, and provenance layer are fully deterministic; no LLM is required to use them.
Quick Start pip install semantica from semantica.context import ContextGraph
graph = ContextGraph(advanced_analytics=True)
Every agent decision becomes a queryable, auditable knowledge node
decision_id = graph.record_decision( category="vendor_selection", scenario="Choose cloud provider for HIPAA workload", reasoning="AWS offers BAA, mature HIPAA tooling, and existing team expertise", outcome="selected_aws", confidence=0.93, )Ask "why did this happen?" and get a real, structured answer
chain = graph.trace_decision_chain(decision_id) # full causal ancestry similar = graph.find_similar_decisions("cloud vendor", max_results=5) # precedents impact = graph.analyze_decision_impact(decision_id) # downstream influence map compliant = graph.check_decision_rules({"category": "vendor_selection"}) # policy gate Verify your install in 5 seconds: semantica doctorPython 3.11.9 pass
semantica 0.6.5 pass
faiss vector store pass
Config file pass ~/.semantica/config.yaml
If Semantica solves a real problem for you, a star helps others find it. ⭐ Star on GitHub · Join Discord
Architecture Semantica is a real end-to-end pipeline, not a single library with a marketing name. Every stage below is a shipping module, independently importable: Sources → Ingest → Parse → Normalize → Split → Extract → Conflict Detection → Deduplication → Knowledge Graph → [ Ontology · Reasoning · Provenance · Decisions ] → Enriched KG → Vector Store + Polyglot Graph Store (RDF & LPG) → Export / Visualize / REST · MCP · CLI
Ingest: files, web, databases, enterprise data platforms (Databricks, Snowflake), cloud (Google Drive, Elasticsearch), streams (Kafka, Kinesis), Git, email, MCP Parse → Normalize → Split: document parsing, text/entity/date normalization, GraphRAG-native entity-aware chunking Extract → Conflict Detection → Deduplication: NER, relations, events, triplets; conflicting facts flagged and resolved before they merge Knowledge Graph: GraphBuilder constructs the graph; bi-temporal facts and full graph analytics (centrality, communities, link prediction) run on top of it Ontology · Reasoning · Provenance · Decisions: the intelligence layer sitting on the KG, with SHACL/OWL governance, Rete/Datalog/SPARQL inference, W3C PROV-O lineage, and first-class decision records Storage: polyglot by design, with RDF triple stores (embedded Oxigraph, Blazegraph, Apache Jena, Eclipse RDF4J), Labeled Property Graphs (Neo4j, FalkorDB, Apache AGE, AWS Neptune), and vector stores, all swappable without touching your code Outputs: export (RDF, OWL, Parquet, Cypher, JSON-LD), interactive visualization, and access via REST API, MCP server, or CLI
→ Full Mermaid diagrams for the pipeline and the decision intelligence lifecycle
Decision Intelligence Decision Intelligence turns every AI choice from an ephemeral inference into a permanent, auditable, queryable record. It answers "what did your AI decide, why, and what happened next?": the question regulators and enterprise risk teams ask with increasing urgency. In Semantica, a decision is not a log line. It is a first-class graph node with a full lifecycle. In regulated domains, every AI decision must be traceable to a source and defensible to an auditor: record_decision() creates a permanent, structured record exportable as W3C PROV-O, the format most compliance frameworks accept for regulator submission. record_decision() → stored as a graph node with full structured context add_causal_relationship() → linked to upstream causes and downstream effects find_similar_decisions() → semantic precedent search across all past decisions trace_decision_chain() → full causal ancestry back to root causes analyze_decision_impact() → downstream influence map - everything this decision affected check_decision_rules() → policy compliance gate against configurable rule sets export / audit trail → W3C PROV-O, CSV, or JSON for regulator submission
from semantica.context import ContextGraph
graph = ContextGraph(advanced_analytics=True)
Record decisions with full structured context
app_id = graph.record_decision( category="credit_application", scenario="Personal loan, $85k income, 31% DTI, 3yr employment", reasoning="Income meets threshold; employment stable; no adverse credit events", outcome="proceed_to_underwriting", confidence=0.88, metadata={"applicant_id": "A-7291"}, ) uw_id = graph.record_decision( category="loan_underwriting", scenario="Underwriting review for A-7291", reasoning="DTI within policy; clean 36-month credit history", outcome="approved", confidence=0.94, ) rate_id = graph.record_decision( category="interest_rate", scenario="Rate assignment for approved loan A-7291", outcome="rate_set_8.9pct", reasoning="Prime + 2.4% based on risk tier B2", confidence=0.99, )Build the auditable causal chain - relationship_type must be one of
CAUSED, INFLUENCED, or PRECEDENT_FOR
graph.add_causal_relationship(app_id, uw_id, relationship_type="CAUSED") graph.add_causal_relationship(uw_id, rate_id, relationship_type="INFLUENCED")Query the intelligence
chain = graph.trace_decision_chain(rate_id) similar = graph.find_similar_decisions("personal loan approval, 31% DTI", max_results=5) impact = graph.analyze_decision_impact(uw_id) compliant = graph.check_decision_rules({"category": "loan_underwriting", "confidence": 0.94}) insights = graph.get_decision_insights()Context Graphs A Context Graph is the structured memory layer that traditional RAG is missing. Instead of flat embeddings that answer "what is similar?", a Context Graph answers "what is connected, why, and how?" Every entity, relationship, decision, and fact is a first-class node, queryable by graph traversal. Entities link to source documents, decisions link to evidence and consequences, facts carry full provenance, and conflicts are detected, not silently overwritten. from semantica.context import ContextGraph, AgentContext from semantica.vector_store import VectorStore
graph = ContextGraph(advanced_analytics=True)
Add nodes with typed properties
graph.add_node("acme_corp", "Organization", name="Acme Corp", industry="SaaS") graph.add_node("alice_chen", "Person", name="Alice Chen", role="CTO") graph.add_node("contract_001", "Contract", value=2_400_000, currency="USD")Add typed, weighted edges (extra kwargs become edge metadata)
graph.add_edge("alice_chen", "acme_corp", edge_type="works_for", since="2019-03-01") graph.add_edge("acme_corp", "contract_001", edge_type="party_to", signed="2024-01-15")BFS traversal - hop through the graph from any node
neighbors = graph.get_neighbors("acme_corp", hops=2)Point-in-time snapshot - the graph as it existed on any past date
snapshot = graph.state_at("2024-01-01")AgentContext - high-level API for agent memory workflows
vs = VectorStore(backend="faiss") ctx = AgentContext(vector_store=vs, knowledge_graph=graph) ctx.store("Alice approved the Acme renewal in Q1 2024", conversation_id="conv_001") retrieved = ctx.retrieve("who approved the Acme contract?") Why graph over embeddings: traversal finds connections embeddings miss (a person 3 hops from a contract); every node carries provenance so you can always ask "where did this come from?"; conflicts are flagged before they corrupt your knowledge base; point-in-time snapshots let you replay history without reprocessing.Recipe: Audit Trail for a Regulated Decision The flagship pattern: record a causally-linked decision chain, attach provenance to every entity, and export a regulator-ready audit trail. from semantica.context import ContextGraph from semantica.provenance import ProvenanceManager from semantica.export import RDFExporter
graph = ContextGraph(advanced_analytics=True) prov = ProvenanceManager(storage_path="./audit.db")
Record the decision chain
d1 = graph.record_decision( category="drug_interaction_check", scenario="Patient P-4821: warfarin + amiodarone co-prescribed", reasoning="Amiodarone potentiates warfarin's anticoagulant effect", outcome="flag_for_review", confidence=0.91, ) d2 = graph.record_decision( category="dosage_adjustment", scenario="INR monitoring plan for P-4821", reasoning="Reduce warfarin dose per interaction severity; recheck INR in 5 days", outcome="dose_reduced_30pct", confidence=0.87, )relationship_type must be one of CAUSED, INFLUENCED, or PRECEDENT_FOR
graph.add_causal_relationship(d1, d2, relationship_type="CAUSED")Track provenance for every entity
prov.track_entity("patient_P4821", source="ehr/medication_orders_2024.json", metadata={"extractor": "NamedEntityRecognizer"})Export W3C PROV-O for regulator submission - to_kg_dict() is the official
adapter that emits the {"entities": [...], "relationships": [...]} /
source_id shape RDFExporter expects, so no manual field mapping is needed
kg = graph.to_kg_dict() RDFExporter().export(kg, "audit_trail.ttl", format="turtle") More recipes (GraphRAG pipelines, an AML rules engine, ontology-to-KG in one pass) are in More Recipes below.Explore the Platform Every module below is independently importable, with working code samples verified against the current source tree; use one or all of them.
Module What it does
semantica.ingest Files, web, databases, APIs, streams, email, Git, Parquet, Databricks, Snowflake, MCP
semantica.semantic_extract NER, relation extraction, event detection, triplet generation
semantica.kg Graph construction, centrality, communities, link prediction
semantica.reasoning Forward chaining, Rete, Datalog, SPARQL, fully explainable
semantica.vector_store FAISS, Qdrant, Weaviate, Milvus, Pinecone, PgVector, hybrid search
semantica.split Entity-aware, relation-aware, ontology-aware chunking for GraphRAG
semantica.provenance W3C PROV-O lineage on every fact
semantica.ontology OWL generation, SHACL validation, SKOS vocabularies
semantica.conflicts Detect and resolve conflicting facts across sources
semantica.deduplication Entity resolution at scale
semantica.normalize Text, entity, date, and number normalization; dataset cleaning
semantica.pipeline Declarative, parallel pipeline DSL for ingest → extract → build → export
semantica.export RDF, OWL, Parquet, Cypher, JSON-LD
semantica.visualization Force-directed graphs, ontology hierarchies, temporal dashboards
Temporal Intelligence Bi-temporal facts, Allen interval algebra, time travel
Multi-Agent (Agno) One shared context graph across every agent on a team
↓ Expand Module Reference below for every module's working example, or jump to More Recipes, the full Integrations matrix, MCP tool list, and REST endpoints.
Module Reference Expand any module below for its runnable example.
semantica.ingest: Multi-Source Ingestion
Ingest from files, web, databases, APIs, streams, email, Git repos, Parquet, Databricks, Snowflake, or MCP servers, all through a unified interface. from semantica.ingest import FileIngestor, WebIngestor, ParquetIngestor, DBIngestor
Ingest an entire directory of contracts (PDF, DOCX, HTML, TXT)
docs = FileIngestor().ingest_directory("./contracts/", recursive=True)Ingest live web content with robots.txt compliance
pages = WebIngestor().ingest_url("https://example.com/reports/annual-2024.html")Ingest structured data from Parquet with Snappy compression
records = ParquetIngestor().ingest("./data/transactions.parquet")Ingest from a SQL database - specify which tables to pull
rows = DBIngestor().ingest_database( connection_string="postgresql://user:pass@localhost/mydb", include_tables=["customer_events"], max_rows_per_table=50_000, )Enterprise data platforms - pull tables straight out of your lakehouse
or warehouse, with lineage, instead of exporting to CSV first
from semantica.ingest import DatabricksIngestor, SnowflakeIngestorpip install "semantica[db-databricks]"
databricks = DatabricksIngestor( host="https://adb-xxx.azuredatabricks.net", token="dapi-xxxxxxxx", # or client_id/client_secret for OAuth M2M http_path="/sql/1.0/warehouses/xxxxxxxx", catalog="main", ) customers = databricks.ingest_table("customers", limit=10_000) sales = databricks.ingest_query("SELECT * FROM sales WHERE region = 'EMEA'") table_lineage = databricks.get_table_lineage("customers", catalog="main", schema="default") # Unity Catalog lineagepip install semantica[db-snowflake]
snowflake = SnowflakeIngestor( account="myaccount", user="myuser", password="mypassword", # or private_key=... for key-pair; use authenticator="oauth", token=... for OAuth warehouse="COMPUTE_WH", database="MYDB", ) orders = snowflake.ingest_table("ORDERS", limit=10_000)Security Note: Never hardcode credentials (token, password, private_key) in production code; pass them via environment variables (e.g., DATABRICKS_TOKEN, SNOWFLAKE_PASSWORD) or a secrets manager.
Supported sources: Local files (PDF, DOCX, PPTX, HTML, TXT, CSV, JSON, YAML, Excel, XML) · Web pages · RSS/Atom feeds · REST APIs · Databases (PostgreSQL, MySQL, SQLite, Oracle, SQL Server) · Parquet datasets · Databricks (Unity Catalog + Delta Lake) · Snowflake · Git repositories · Email (IMAP/POP3) · Message streams (Kafka, RabbitMQ, Kinesis, Pulsar) · MCP resources · Apache Arrow/Feather/IPC (ArrowIngestor) DuckDB, Elasticsearch, Google Drive, HuggingFace, MongoDB, and Pandas ingestion also ship (DuckDBIngestor, ElasticIngestor, GDriveIngestor, HuggingFaceIngestor, MongoIngestor, PandasIngestor) but aren't re-exported from the top-level semantica.ingest namespace yet — import them directly: from semantica.ingest.duckdb_ingestor import DuckDBIngestor.
semantica.semantic_extract: NER, Relations, Events, Triplets
Extract structured knowledge from raw text in one pass. from seman