The Missing Web UI for Elasticsearch: Import, browse and edit data with rich filters and query views, create search UIs visually.

  • By appbase.io
  • Last update: Sep 22, 2022
  • Comments: 16

dejavu: The missing Web UI for Elasticsearch

GitHub License React Version Docker Pulls

  1. Dejavu: Intro
  2. Features
    a. Easily Connect to Indices
    b. Visual Filters
    c. Modern UI Elements
    d. Import JSON or CSV Data
    e. Build search UIs
  3. Comparison
  4. Roadmap
  5. Build Locally / Contributing
  6. Get Dejavu
    a. Docker Installation
    b. Hosted Alternatives

1. Dejavu Intro

dejavu is the missing web UI for Elasticsearch. Existing web UIs leave much to be desired or are built with server-side page rendering techniques that make it less responsive and bulkier to run (I am looking at you, Kibana).

We started building dejavu with the goal of creating a modern Web UI (no page reloads, infinite scroll, filtered views, realtime updates, search UI builder) for Elasticsearch with 100% client-side rendering so one can easily run it as a hosted app on github pages, as a chrome extension or as a docker image.

Starting v1.0, dejavu is the only Elasticsearch web UI that supports importing data via JSON and CSV files, as well as defining field mappings from the GUI.

Starting with v1.5, we support the ability of creating custom headers so you can easily pass different authentication headers, provide enhanced filtering and bulk updating of data via Elasticsearch's Query DSL.

Starting with v2.0, we support the ability to build faceted search UIs to test relevancy. You can also export the generated code to a codesandbox.

Starting with v3.0, we support the ability to connect to multiple indexes. You can also globally search across your indexes using global search bar.

2. Features

Easily Connect and Remember Indices

Connect to an Index

Dejavu allows you to connect to any of the indexes present in your cluster and also caches each connected index locally so they are easily accessible when browsing again.

Visual Filters

Filter Views

Sort through the data, find information visually, hide irrelevant data and make sense of all. With all the native data types we have . Global searchbar allows you to perform text search across your dataset.

Moreover, any filtered view can be exported as a JSON or CSV file.

Modern UI elements

Pagination

It's not uncommon to have thousands of documents in your index. Dejavu supports paginated view which also allows you to change page size.

Dejavu also supports browsing data from multiple indexes and types, updating data either individually or via queries in bulk. Deletions are also supported.

Import JSON or CSV Data

Import JSON or CSV files

Importer view allows importing CSV or JSON data directly into Elasticsearch through a guided data mappings configuration.

Build Search UIs

Build search UIs

With Search Preview, you can now build visual search UIs, test search relevancy and export code to a codesandbox.


3. Comparison with other data browsers

Features dejavu ES-head ES-kopf ES-browser Kibana
Installation Chrome extension, Docker image, Hosted app Elasticsearch plugin, static page Elasticsearch plugin, static page Elasticsearch plugin (doesn't work with 2.0+) Elasticsearch plugin
Modern UI React 16.6. jQuery 1.6.1, slightly stodgy Angular 1.x ExtJs, a bit stodgy Node.JS, Hapi, Jade
Browser features CRUD, data filters Read data, full-text search Data view for a single type Read view, visualizations, charting
Data import/export ✔️ JSON, CSV Only export, no CSV
Search preview Visually build and test search UI
License MIT Apache 2.0 MIT Apache 2.0 Apache 2.0

4. Roadmap

Here's a rough roadmap of things to come in the version 1.0.0 release.

🎆 We just hit the 1.0.0 roadmap:

  • Battle-testing with different datasets
  • Feature support for advanced filtering Offline detection and reconnection for realtime updates
  • Performance improvements while scrolling
  • Support for importing and exporting data
  • Support for a continuous query view
  • Available as a docker image

🍾 We just hit the 2.0.0 release:

  • An intuitive data editing experience in tabular mode (v/s JSON edit mode)
  • View data types from within the data browser view
  • A more streamlined import process
  • Refactor codebase to improve hackability (Migrate to React 16+, ES6 syntax)
  • Ability to build (and test) search visually

We just hit the 3.0.0 release:

  • Rewrite dejavu browser for high performance when browsing large datasets
  • Add support for browsing multiple indexes
  • Powerful filtering of data with field level facet based filters and a global search
  • Built on React 16.6 and future compatible with React 17
  • A more intuitive data editing experience (in addition to the raw JSON, we now show a relevant UI field with validations)

🔜 Roadmap beyond 3.0.0:

  • Improve test coverage
  • Mobile friendly viewing and editing experience
  • Additional filtering support (add range filters, date filters)

5. Build Locally

See the contributing guidelines.


6. Get Dejavu

Docker Installation

docker run -p 1358:1358 -d appbaseio/dejavu
open http://localhost:1358/

You can also run a specific version of dejavu by specifying a tag. For example, version 3.3.0 can be used by specifying the docker run -p 1358:1358 appbaseio/dejavu:3.3.0 command.

Cross-origin resource sharing (CORS)

To make sure you enable CORS settings for your Elasticsearch instance, add the following lines in the elasticsearch.yml configuration file.

http.port: 9200
http.cors.allow-origin: 'http://localhost:1358'
http.cors.enabled: true
http.cors.allow-headers: X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: true

If you are running your Elasticsearch with docker-compose, you can refer to the example reference here.

If you are running your Elasticsearch with docker, you can use the following flags to pass the custom CORS configuration:

Elasticsearch 2.x
docker run --name elasticsearch -p 9200:9200 -d elasticsearch:2 -Des.http.port=9200 -Des.http.cors.allow-origin="http://localhost:1358" -Des.http.cors.enabled=true -Des.http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization -Des.http.cors.allow-credentials=true
Elasticsearch 5.x
docker run --name elasticsearch -p 9200:9200 -d elasticsearch:5 -E http.port=9200 -E http.cors.allow-origin="http://localhost:1358" -E http.cors.enabled=true -E http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization -E http.cors.allow-credentials=true
Elasticsearch 6.x
docker run -p 9200:9200 -d elasticsearch docker.elastic.co/elasticsearch/elasticsearch-oss:6.5.4 -Ehttp.port=9200 -Ehttp.cors.enabled=true -Ehttp.cors.allow-origin=http://localhost:1358 -Ehttp.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization -Ehttp.cors.allow-credentials=true
Elasticsearch 7.x
docker run -d --rm --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "http.cors.enabled=true" -e "http.cors.allow-origin=*" -e "http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization" -e "http.cors.allow-credentials=true" docker.elastic.co/elasticsearch/elasticsearch-oss:7.0.1

Hosted Alternatives

dejavu can also be run via hosted app at https://dejavu.appbase.io or installed as a chrome extension.

For example: If you are using the chrome-extension instead of docker image, the http.cors.allow-origin in Elasticsearch.yml file would change accordingly:

http.port: 9200
http.cors.allow-origin: 'chrome-extension://jopjeaiilkcibeohjdmejhoifenbnmlh'
http.cors.enabled: true
http.cors.allow-headers: X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization
http.cors.allow-credentials: true

Github

https://github.com/appbaseio/dejavu

Comments(16)

  • 1

    Can't connect to elasticsearch

    Have installed via Docker Can't connect to elasticsearch instance My elasticsearch installation is on a separate machine. when logging in, loading screen just loops.

  • 2

    Cannot connect using username and password

    I can't connect to ES using my username and password when xpack.security.enabled is set to true.

    My URL format in dejavu is like: https://username:[email protected]:9200

    My index: *

    My result:

    360截图20190727160114102

    I can connect to ES in Postman manually using https://username:[email protected]:9200

  • 3

    Connecting is a bit unintuitive...

    When starting dejavu (Chrome Extension) for the first time and it's not clear what we need to do. When we enter the URL http://myserver:9200 nothing happens. It took several confused minutes to realise that, until an appname is entered, the connect button is disabled. It's obvious once you've done it but one idea might be to leave the button enabled (and put it on the left) and post a warning when it is clicked (or Enter is pressed) about missing connection information.

    Alternatively, after the URL is entered/changed you could actually load the list of indices as a dropdown box.

    image

  • 4

    CORS issues when running using docker-compose

    When I run Elasticsearch and Dejavu using docker-compose, I get CORS errors even though I configured it in the docker-compose.yml.

    docker-compose.yml

    version: '3'
    services:
      elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:6.5.2
        container_name: elasticsearch
        environment:
          - discovery.type=single-node
          - cluster.name=docker-cluster
          - bootstrap.memory_lock=true
          - transport.host=127.0.0.1
          - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
          - http.host=0.0.0.0
          - http.port=9200
          - "http.cors.allow-origin=http://localhost:1358"
          - "http.cors.enabled=true"
          - "http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization"
          - "http.cors.allow-credentials=true"
        ulimits:
          memlock:
            soft: -1
            hard: -1
        volumes:
          - esdata:/usr/share/elasticsearch/data
        ports:
          - "9200:9200"
          - "9300:9300"
        networks:
          - esnet
    
      kibana:
        image: docker.elastic.co/kibana/kibana-oss:6.5.2
        container_name: kibana
        environment:
          SERVER_NAME: kibana-server
          ELASTICSEARCH_URL: http://elasticsearch:9200
        networks:
          - esnet
        depends_on:
          - elasticsearch
        ports:
          - "5601:5601"
    
      cerebro:
        image: yannart/cerebro:0.8.1
        container_name: cerebro
        networks:
          - esnet
        ports:
          - "9900:9000"
        depends_on:
          - elasticsearch
    
      dejavu:
        image: appbaseio/dejavu:2.0.5
        container_name: dejavu
        ports:
          - "1358:1358"
        networks:
          - esnet
        links:
          - elasticsearch
    
    volumes:
      esdata:
        driver: local
    
    networks:
      esnet:
        driver: bridge
    
  • 5

    Define Index Patterns

    I'm just trying Chrome extension version 1.6.1, and I use elasticsearch as a primary log sink. As most of log handler will create index daily, so this is very import to define index patterns.

    Maybe I miss something, but I can't find anything about this feature. Would someone give me some clues about it?

  • 6

    Dejavu / importer - JSON import not working for web app

    Tell us where you are using Dejavu (Hosted web app, Chrome extension, Docker Image, Within appbase.io)

    Hosted web app

    What is the version of Dejavu that you are using?

    N/A

    Describe the issue that you are seeing, or the feature request :-), include any screenshots as necessary.

    I upload a JSON file using the GUI. After configuring the data, I click the "Start Import" button, and then the "Start" button. After this nothing happens. When I query elasticsearch, it shows the index was created, but the documents within the JSON are missing within the index. See attached video and JSON file for reference.

    screen recording products_small.txt

    **Note that I can copy the data from the JSON file into the index while in "Editing" mode using the "Insert JSON" button, and all the documents are uploaded fine.

    If your issue deals with accessing Elasticsearch cluster, share the necessary steps to replicate the behavior (the configuration.yml settings, where your ES cluster is hosted, any relevant logs).
    If your issue deals with a UI issue, share with us a screenshot of the failing network request or the browser console log showing the error.
  • 7

    Cannot read property 'properties' of undefined on DataTable.js after login

    Tell us where you are using Dejavu (Hosted web app, Chrome extension, Docker Image, Within appbase.io)

    When running dejavu using the cloned git dev branch with ES 2.4. npm start

    What is the version of Dejavu that you are using?

    latest dev branch as of now

    Describe the issue that you are seeing, or the feature request :-), include any screenshots as necessary.

    After key in the Appname(index) and the url (which is http://localhost:9200 with CORS enabled) It logins and loaded the mappings, then a react error screen appear with the errors. TypeError: Cannot read property 'properties' of undefined properties /Users/leng/Documents/library/dejavu/live/src/js/table/DataTable.js DataTable.tryRender webpack:///~/react-transform-catch-errors/lib/index.js:34 DataTable.proxiedMethod webpack:///~/react-proxy/modules/createPrototypeProxy.js:44 ReactCompositeComponentWrapper._renderValidatedComponentWithoutOwnerOrContext webpack:///~/react/lib/ReactCompositeComponent.js:587 ReactCompositeComponentWrapper._renderValidatedComponent webpack:///~/react/lib/ReactCompositeComponent.js:607 ReactCompositeComponentWrapper.wrapper webpack:///~/react/lib/ReactPerf.js:66 ReactCompositeComponentWrapper._updateRenderedComponent webpack:///~/react/lib/ReactCompositeComponent.js:560 ReactCompositeComponentWrapper._performComponentUpdate webpack:///~/react/lib/ReactCompositeComponent.js:544 ReactCompositeComponentWrapper.updateComponent webpack:///~/react/lib/ReactCompositeComponent.js:473 ReactCompositeComponentWrapper.wrapper webpack:///~/react/lib/ReactPerf.js:66

    If your issue deals with a UI issue, share with us a screenshot of the failing network request or the browser console log showing the error.

    Network log is all okay with status 200. Js console errors as attached. wechatimg50

  • 8

    importer can't work

    I have install dejavu through docker in my server, while when I want use import feature, it show me this page, I tried to login by github or google, but all fail.

    console print XMLHttpRequest cannot load https://accapi.appbase.io/user. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://<mydomain>:1358' is therefore not allowed access.

  • 9

    Unable to connect

    Hello To be very brief ! I can't connect ! This is my Docker Compos configuration When I enter the elastic address(http://0.0.0.0:9200), it does not connect... In addition, I see my indexes easily in Kibana

    es7: image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.10.2 logging: driver: none container_name: c-es environment: - discovery.type=single-node - http.port=9200 - http.cors.enabled=true - http.cors.allow-origin=http://dejavu:1358,http://dejavu:1358 - http.cors.allow-headers=X-Requested-With,X-Auth-Token,Content-Type,Content-Length,Authorization - http.cors.allow-credentials=true - bootstrap.memory_lock=true - 'ES_JAVA_OPTS=-Xms512m -Xmx512m' ports: - "9200:9200" networks: - test ulimits: memlock: soft: -1 hard: -1 restart: unless-stopped

    dejavu: image: appbaseio/dejavu:3.6.0 container_name: dejavu ports: - '1358:1358' links: - es7 networks: - test

  • 10

    Filling`App name` and `URL` entries

    I am running both Elasticsearch and Dejavu on a remote machine and am connecting to Dejavu by port forwarding port 1358 over ssh. When I open http://localhost:1358 I get to see Dejavu UI. Here's what I entered in these fields: App: <my_index_name> (confirmed that this index exists in ES) URL: http://localhost:9200

    After this Dejavu fails to connect and hangs

    1. Are these settings right? -If yes then what else could I try?

    I have verified that I can connect to the ES instance using http://localhost:9200 by forwarding port 9200

  • 11

    Can 'input_state' potentially save an actual query as well as the client, index and types contexts?

    Thanks for this project and its sister mirage. Both serve an important gap in the ES GUI space.

    I've been playing around with a local installation against my data and trying to get a better feel for how 'input_state' is working, in terms of it somehow encoding a 'state' ... ie. client-connection info, index selection and perhaps type selection. I don't think it's base64 encoded ... but I can see you are using cryptojs in some way.

    Are there any spec documents as to how this is working ... or perhaps an example of an encoded/decoded payload.

    I was also wondering if it's possible to include a query that gets applied to the given 'state', and pass that as part of the url ... in either encoded or plain JSON form?

    Thanks.

  • 12

    Eliminate document properties from search results which are null or blank

    Is your feature request related to a problem? Please describe. I am playing around with Dejavu and many of my document properties are null or blank.

    Describe the solution you'd like When i click on the search button via the search preview I don't want the results to include properties from the doc where the properties are blank or null,

    Describe alternatives you've considered I haven't considered any alternative so far.

    Additional context Lets say i have a document like this {"firstName":"Asfand", "lastName":"Hikmat","Location":"UK", "salary":null}, {"firstName":"Maria", "lastName":"ZaiZai", "location":"UK", "salary":"2000$"

    When i search for this document in the search UI for UK can the result set include something like this. "firstName":"Asfand", "lastName":"Hikmat","Location":"UK" "firstName":"Maria", "lastName":"ZaiZai", "location":"UK", "salary":"2000$"

    Note for the first doc since salary is null we don't show it in the search UI.

  • 13

    Failure to propogate signals to Docker container

    Describe the bug When running dejavu in a docker container in the foreground, SIGINT and SIGTERM signals are not propagating to the process.

    Dejavu Version Docker image: latest (as of 4/11/22)

    To Reproduce Steps to reproduce the behavior:

    1. Run docker run -p 1358:1358 --rm appbaseio/dejavu
    2. Press Ctrl+C

    Expected behavior The container should exit after pressing Ctrl+C

    Desktop (please complete the following information):

    • OS: MacOS 12.3.1

    Additional context This is likely due to not using the exec syntax when running the CMD for node. This is a good article explaining it https://hynek.me/articles/docker-signals/ The likely fix is to use this form CMD ["node", "packages/dejavu-main/server.js"] in the Dockerfile. This is important because it enables any program that orchestrates running multiple processes (for example running elasticsearch, dejavu and and app) to safely kill this running container at shutdown time. It also allows the docker stop command to safely stop the dejavu server without using a force kill.

    I ran into a yarn build error when trying to run docker build locally otherwise I would have put up a PR with the fix.

  • 14

    Want to achieve automatic cross domain

    Is your feature request related to a problem? Please describe.

    When using the company's es, you cannot modify the ES configuration. At this time, it's good to have automatic cross domain Describe the solution you'd like A clear and concise description of what you want to happen.

    refer to:https://github.com/qax-os/ElasticHD There is no need to modify the configuration

  • 15

    Babel Error; ARM

    See: dejavu on ARM64 docker #411

    Apple Silicon M1 Pro Laptop; running a Debian Virtual machine under Parallels. CPU arch of the VM below: uname -r 5.10.0-10-arm64

    cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

    With QEMU installed; I am able to run the X86 binaries without issue from the docker-hub pull.

    docker pull appbaseio/dejavu docker run -p 1358:1358 -d appbaseio/dejavu

    and when viewing in docker; confirms image is X86 Build | Docker on linux, amd64 -- | -- Author | appbase.io [email protected]

    Issue is specific when trying to pull from source and build under arm binaries

    Exact steps run: git clone -b fix/babel-optional-chaining https://github.com/appbaseio/dejavu.git cd dejavu git submodule init && git submodule update --recursive --remote docker build -t dejavu:arm .

  • 16

    Missing .js file ERROR 404

    I am trying to run dejavu inside a docker container (swarm) running behind an Nginx. on loading the page I get the title: Capture d’écran 2022-01-04 à 10 58 15

    But nothing renders on the screen, And I get a few errors Capture d’écran 2022-01-04 à 10 48 44

    I am using the latest version of dejavu Tried with safari firefox and chrome...

    If anyone has an idea of how to solve this.