Datawave is a Java-based ingest and query framework that leverages Apache Accumulo to provide fast, secure access to your data. Datawave supports a wide variety of use cases, including but not limited to:
- Data fusion across structured and unstructured datasets
- Construction and analysis of distributed graphs
- Multi-tenant data architectures, with tenants having distinct security requirements and data access patterns
- Fine-grained control over data access, integrated easily with existing user-authorization services and PKI
Below is the basic structure of Datawave:
Here's a graphic I made that get's a little more in-depth:
Now that you know how the architecture flows a bit better let's start with installing Datawave, then we'll get into Edges.
Getting started with NSA's Datawave
Before we start, NB: You should have an understanding of using simple Bash scripting, Linux commands like grep awk and using piping.
What you'll need
Linux, Bash, and an Internet connection to wget tarballs (tar.gz)that you should be able to ssh to localhost without a passphrase.
Note that the quickstart Hadoop install will set up passphrase-less ssh for you automatically, unless it detects that you already have a private/public key pair generated
Familiarize yourself with swap and/or swapping if you haven't already. You'll need this. https://wiki.gentoo.org/wiki/Swap
Installing Datawave through the CLI in 5 commands
echo "source DW_SOURCE/contrib/datawave-quickstart/bin/env.sh" >> ~/.bashrc
source ~/.bashrc
allInstall
datawaveWebStart && datawaveWebTest
So we're adding sources to our .bashrc this can be true too if you're using zsh.
The four commands above will complete the entire quickstart installation. However, it’s a good idea to at least skim over the sections below to get an idea of how the setup works and how to customize it for your own preferences.
To keep things simple, DataWave, Hadoop, Accumulo, ZooKeeper, and Wildfly will be installed under your DW_SOURCE/contrib/datawave-quickstart directory, and all will be owned by / executed as the current user, hence why a bash script in the background was being ran.
Overriding your default binaries
On some occasions you may need to override the default binaries (not in all machines, or setups). Let's open up Vim, and do this in case you do need to end up overriding your binaries. To override the quickstarts default version of a particular binary, simply override the desired DW_*_DIST_URI value as shown below. URIs may be local or remote. Local file URI values must be prefixed with file://, so let's start:
vi ~/.bashrc
export DW_HADOOP_DIST_URI=file:///my/local/binaries/hadoop-x.y.z.tar.gz
export DW_ACCUMULO_DIST_URI=http://some.apache.mirror/accumulo/1.x/accumulo-1.x-bin.tar.gz
export DW_ZOOKEEPER_DIST_URI=http://some.apache.mirror/zookeeper/x.y/zookeeper-x.y.z.tar.gz
export DW_WILDFLY_DIST_URI=file:///my/local/binaries/wildfly-10.x.tar.gz
export DW_JAVA_DIST_URI=file:///my/local/binaries/jdk-8-update-x.tar.gz
export DW_MAVEN_DIST_URI=file:///my/local/binaries/apache-maven-x.y.z.tar.gz
We just grabbed Apache Hadoop, Accumulo, Zookeeper, Wildfly, and Maven. Now if this seems like a lot, we can always bootstrap your environment with a bash script, this doesn't give you as much flexibility (unless you want to add it), but it is quicker, add your own shebang line at the top. Make sure you make this bash script executable, when done copy/pasting, run chmod u+x bootstrap_datawave.sh:
DW_DATAWAVE_SERVICE_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
DW_DATAWAVE_SOURCE_DIR="$( cd "${DW_DATAWAVE_SERVICE_DIR}/../../../../.." && pwd )"
DW_DATAWAVE_ACCUMULO_AUTHS="${DW_DATAWAVE_ACCUMULO_AUTHS:-PUBLIC,PRIVATE,FOO,BAR,DEF,A,B,C,D,E,F,G,H,I,DW_USER,DW_SERV,DW_ADMIN,JBOSS_ADMIN}"
# Import DataWave Web test user configuration
source "${DW_DATAWAVE_SERVICE_DIR}/bootstrap-user.sh"
# Selected Maven profile for the DataWave build
DW_DATAWAVE_BUILD_PROFILE=${DW_DATAWAVE_BUILD_PROFILE:-dev}
# Maven command
DW_DATAWAVE_BUILD_COMMAND="${DW_DATAWAVE_BUILD_COMMAND:-mvn -P${DW_DATAWAVE_BUILD_PROFILE} -Ddeploy -Dtar -Ddist -Dservices -DskipTests clean install --builder smart -T1.0C}"
# Home of any temp data and *.properties file overrides for this instance of DataWave
DW_DATAWAVE_DATA_DIR="${DW_CLOUD_DATA}/datawave"
# Temp dir for persisting our dynamically-generated ${DW_DATAWAVE_BUILD_PROFILE}.properties file
DW_DATAWAVE_BUILD_PROPERTIES_DIR="${DW_DATAWAVE_DATA_DIR}/build-properties"
DW_DATAWAVE_BUILD_STATUS_LOG="${DW_DATAWAVE_BUILD_PROPERTIES_DIR}/build-progress.tmp"
DW_DATAWAVE_INGEST_TARBALL="*/datawave-${DW_DATAWAVE_BUILD_PROFILE}-*-dist.tar.gz"
DW_DATAWAVE_WEB_TARBALL="*/datawave-ws-deploy-application-*-${DW_DATAWAVE_BUILD_PROFILE}.tar.gz"
DW_DATAWAVE_KEYSTORE="${DW_DATAWAVE_KEYSTORE:-${DW_DATAWAVE_SOURCE_DIR}/web-services/deploy/application/src/main/wildfly/overlay/standalone/configuration/certificates/testServer.p12}"
DW_DATAWAVE_KEYSTORE_PASSWORD=${DW_DATAWAVE_KEYSTORE_PASSWORD:-ChangeIt}
DW_DATAWAVE_KEYSTORE_TYPE="${DW_DATAWAVE_KEYSTORE_TYPE:-PKCS12}"
DW_DATAWAVE_TRUSTSTORE="${DW_DATAWAVE_TRUSTSTORE:-${DW_DATAWAVE_SOURCE_DIR}/web-services/deploy/application/src/main/wildfly/overlay/standalone/configuration/certificates/ca.jks}"
DW_DATAWAVE_TRUSTSTORE_PASSWORD=${DW_DATAWAVE_TRUSTSTORE_PASSWORD:-ChangeIt}
DW_DATAWAVE_TRUSTSTORE_TYPE="${DW_DATAWAVE_TRUSTSTORE_TYPE:-JKS}"
# Accumulo shell script for initializing whatever we may need in Accumulo for DataWave
function createAccumuloShellInitScript() {
# Allow user to inject their own script into the env...
[ -n "${DW_ACCUMULO_SHELL_INIT_SCRIPT}" ] && return 0
# Create script and add 'datawave' VFS context, if enabled...
DW_ACCUMULO_SHELL_INIT_SCRIPT="
createnamespace datawave
createtable datawave.queryMetrics_m
createtable datawave.queryMetrics_s
setauths -s ${DW_DATAWAVE_ACCUMULO_AUTHS}"
if [ "${DW_ACCUMULO_VFS_DATAWAVE_ENABLED}" != false ] ; then
DW_ACCUMULO_SHELL_INIT_SCRIPT="${DW_ACCUMULO_SHELL_INIT_SCRIPT}
config -s table.classpath.context=datawave"
fi
DW_ACCUMULO_SHELL_INIT_SCRIPT="${DW_ACCUMULO_SHELL_INIT_SCRIPT}
quit
"
}
function createBuildPropertiesDirectory() {
if [ ! -d ${DW_DATAWAVE_BUILD_PROPERTIES_DIR} ] ; then
if ! mkdir -p ${DW_DATAWAVE_BUILD_PROPERTIES_DIR} ; then
error "Failed to create directory ${DW_DATAWAVE_BUILD_PROPERTIES_DIR}"
return 1
fi
fi
return 0
}
function setBuildPropertyOverrides() {
# DataWave's build configs (*.properties) can be loaded from a variety of locations based on the 'read-properties'
# Maven plugin configuration. Typically, the source-root/properties/*.properties files are loaded first to provide
# default values, starting with 'default.properties', followed by '{selected-profile}.properties'. Finally,
# ~/.m2/datawave/properties/{selected-profile}.properties is loaded, if it exists, allowing you to override
# defaults as needed
# With that in mind, the goal of this function is to generate a new '${DW_DATAWAVE_BUILD_PROFILE}.properties' file under
# DW_DATAWAVE_BUILD_PROPERTIES_DIR and *symlinked* as ~/.m2/datawave/properties/${DW_DATAWAVE_BUILD_PROFILE}.properties,
# to inject all the overrides that we need for successful deployment to source-root/contrib/datawave-quickstart/
# If a file having the name '${DW_DATAWAVE_BUILD_PROFILE}.properties' already exists under ~/.m2/datawave/properties,
# then it will be renamed automatically with a ".saved-by-quickstart-$(date)" suffix, and the symlink for the new
# file will be created as required
local BUILD_PROPERTIES_BASENAME=${DW_DATAWAVE_BUILD_PROFILE}.properties
local BUILD_PROPERTIES_FILE=${DW_DATAWAVE_BUILD_PROPERTIES_DIR}/${BUILD_PROPERTIES_BASENAME}
local BUILD_PROPERTIES_SYMLINK_DIR=${HOME}/.m2/datawave/properties
local BUILD_PROPERTIES_SYMLINK=${BUILD_PROPERTIES_SYMLINK_DIR}/${BUILD_PROPERTIES_BASENAME}
! createBuildPropertiesDirectory && error "Failed to override properties!" && return 1
# Create symlink directory if it doesn't exist
[ ! -d ${BUILD_PROPERTIES_SYMLINK_DIR} ] \
&& ! mkdir -p ${BUILD_PROPERTIES_SYMLINK_DIR} \
&& error "Failed to create symlink directory ${BUILD_PROPERTIES_SYMLINK_DIR}" \
&& return 1
# Copy existing source-root/properties/${DW_DATAWAVE_BUILD_PROFILE}.properties to our new $BUILD_PROPERTIES_FILE
! cp "${DW_DATAWAVE_SOURCE_DIR}/properties/${DW_DATAWAVE_BUILD_PROFILE}.properties" ${BUILD_PROPERTIES_FILE} \
&& error "Aborting property overrides! Failed to copy ${DW_DATAWAVE_BUILD_PROFILE}.properties" \
&& return 1
# Apply overrides as needed by simply appending them to the end of the file...
echo "#" >> ${BUILD_PROPERTIES_FILE}
echo "######## Begin overrides for datawave-quickstart ########" >> ${BUILD_PROPERTIES_FILE}
echo "#" >> ${BUILD_PROPERTIES_FILE}
echo "WAREHOUSE_ACCUMULO_HOME=${ACCUMULO_HOME}" >> ${BUILD_PROPERTIES_FILE}
echo "WAREHOUSE_INSTANCE_NAME=${DW_ACCUMULO_INSTANCE_NAME}" >> ${BUILD_PROPERTIES_FILE}
echo "WAREHOUSE_JOBTRACKER_NODE=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
echo "INGEST_ACCUMULO_HOME=${ACCUMULO_HOME}" >> ${BUILD_PROPERTIES_FILE}
echo "INGEST_INSTANCE_NAME=${DW_ACCUMULO_INSTANCE_NAME}" >> ${BUILD_PROPERTIES_FILE}
echo "INGEST_JOBTRACKER_NODE=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
echo "BULK_INGEST_DATA_TYPES=${DW_DATAWAVE_INGEST_BULK_DATA_TYPES}" >> ${BUILD_PROPERTIES_FILE}
echo "LIVE_INGEST_DATA_TYPES=${DW_DATAWAVE_INGEST_LIVE_DATA_TYPES}" >> ${BUILD_PROPERTIES_FILE}
echo "PASSWORD=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "ZOOKEEPER_HOME=${ZOOKEEPER_HOME}" >> ${BUILD_PROPERTIES_FILE}
echo "HADOOP_HOME=${HADOOP_HOME}" >> ${BUILD_PROPERTIES_FILE}
echo "MAPRED_HOME=${HADOOP_HOME}" >> ${BUILD_PROPERTIES_FILE}
echo "WAREHOUSE_HADOOP_CONF=${HADOOP_CONF_DIR}" >> ${BUILD_PROPERTIES_FILE}
echo "INGEST_HADOOP_CONF=${HADOOP_CONF_DIR}" >> ${BUILD_PROPERTIES_FILE}
echo "HDFS_BASE_DIR=${DW_DATAWAVE_INGEST_HDFS_BASEDIR}" >> ${BUILD_PROPERTIES_FILE}
echo "MAPRED_INGEST_OPTS=${DW_DATAWAVE_MAPRED_INGEST_OPTS}" >> ${BUILD_PROPERTIES_FILE}
echo "LOG_DIR=${DW_DATAWAVE_INGEST_LOG_DIR}" >> ${BUILD_PROPERTIES_FILE}
echo "FLAG_DIR=${DW_DATAWAVE_INGEST_FLAGFILE_DIR}" >> ${BUILD_PROPERTIES_FILE}
echo "FLAG_MAKER_CONFIG=${DW_DATAWAVE_INGEST_FLAGMAKER_CONFIGS}" >> ${BUILD_PROPERTIES_FILE}
echo "BIN_DIR_FOR_FLAGS=${DW_DATAWAVE_INGEST_HOME}/bin" >> ${BUILD_PROPERTIES_FILE}
echo "KEYSTORE=${DW_DATAWAVE_KEYSTORE}" >> ${BUILD_PROPERTIES_FILE}
echo "KEYSTORE_TYPE=${DW_DATAWAVE_KEYSTORE_TYPE}" >> ${BUILD_PROPERTIES_FILE}
echo "KEYSTORE_PASSWORD=${DW_DATAWAVE_KEYSTORE_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "TRUSTSTORE=${DW_DATAWAVE_TRUSTSTORE}" >> ${BUILD_PROPERTIES_FILE}
echo "TRUSTSTORE_PASSWORD=${DW_DATAWAVE_TRUSTSTORE_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "TRUSTSTORE_TYPE=${DW_DATAWAVE_TRUSTSTORE_TYPE}" >> ${BUILD_PROPERTIES_FILE}
echo "FLAG_METRICS_DIR=${DW_DATAWAVE_INGEST_FLAGMETRICS_DIR}" >> ${BUILD_PROPERTIES_FILE}
echo "accumulo.instance.name=${DW_ACCUMULO_INSTANCE_NAME}" >> ${BUILD_PROPERTIES_FILE}
echo "accumulo.user.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "cached.results.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
echo "type.metadata.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
echo "mapReduce.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
echo "bulkResults.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
echo "jboss.log.hdfs.uri=${DW_HADOOP_DFS_URI}" >> ${BUILD_PROPERTIES_FILE}
echo "lock.file.dir=${DW_DATAWAVE_INGEST_LOCKFILE_DIR}" >> ${BUILD_PROPERTIES_FILE}
echo "server.keystore.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "mysql.user.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "jboss.jmx.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "jboss.managed.executor.service.default.max.threads=${DW_WILDFLY_EE_DEFAULT_MAX_THREADS:-48}" >> ${BUILD_PROPERTIES_FILE}
echo "hornetq.cluster.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "hornetq.system.password=${DW_ACCUMULO_PASSWORD}" >> ${BUILD_PROPERTIES_FILE}
echo "mapReduce.job.tracker=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
echo "bulkResults.job.tracker=${DW_HADOOP_RESOURCE_MANAGER_ADDRESS}" >> ${BUILD_PROPERTIES_FILE}
echo "EVENT_DISCARD_INTERVAL=0" >> ${BUILD_PROPERTIES_FILE}
echo "ingest.data.types=${DW_DATAWAVE_INGEST_LIVE_DATA_TYPES},${DW_DATAWAVE_INGEST_BULK_DATA_TYPES}" >> ${BUILD_PROPERTIES_FILE}
echo "JOB_CACHE_REPLICATION=1" >> ${BUILD_PROPERTIES_FILE}
echo "EDGE_DEFINITION_FILE=${DW_DATAWAVE_INGEST_EDGE_DEFINITIONS}" >> ${BUILD_PROPERTIES_FILE}
echo "DATAWAVE_INGEST_HOME=${DW_DATAWAVE_INGEST_HOME}" >> ${BUILD_PROPERTIES_FILE}
echo "PASSWORD_INGEST_ENV=${DW_DATAWAVE_INGEST_PASSWD_FILE}" >> ${BUILD_PROPERTIES_FILE}
echo "hdfs.site.config.urls=file://${HADOOP_CONF_DIR}/core-site.xml,file://${HADOOP_CONF_DIR}/hdfs-site.xml" >> ${BUILD_PROPERTIES_FILE}
echo "table.shard.numShardsPerDay=${DW_DATAWAVE_INGEST_NUM_SHARDS}" >> ${BUILD_PROPERTIES_FILE}
generateTestDatawaveUserServiceConfig
# Apply DW_JAVA_HOME_OVERRIDE, if needed...
# We can override the JAVA_HOME location for the DataWave deployment, if necessary. E.g., if we're deploying
# to a Docker container or other, where our current JAVA_HOME isn't applicable
if [ -n "${DW_JAVA_HOME_OVERRIDE}" ] ; then
echo "JAVA_HOME=${DW_JAVA_HOME_OVERRIDE}" >> ${BUILD_PROPERTIES_FILE}
else
echo "JAVA_HOME=${JAVA_HOME}" >> ${BUILD_PROPERTIES_FILE}
fi
# Apply DW_ROOT_DIRECTORY_OVERRIDE, if needed...
# We can override any instances of DW_DATAWAVE_SOURCE_DIR within the build config in order to relocate
# the deployment, if necessary. E.g., used when building the datawave-quickstart Docker image to reorient
# the deployment under /opt/datawave/ within the container
if [ -n "${DW_ROOT_DIRECTORY_OVERRIDE}" ] ; then
sed -i "s~${DW_DATAWAVE_SOURCE_DIR}~${DW_ROOT_DIRECTORY_OVERRIDE}~g" ${BUILD_PROPERTIES_FILE}
fi
# Create the symlink under ~/.m2/datawave/properties
setBuildPropertiesSymlink || return 1
}
function setBuildPropertiesSymlink() {
# Replace any existing ~/.m2/datawave/properties/${BUILD_PROPERTIES_BASENAME} file/symlink with
# a symlink to our new ${BUILD_PROPERTIES_FILE}
if [[ -f ${BUILD_PROPERTIES_SYMLINK} || -L ${BUILD_PROPERTIES_SYMLINK} ]] ; then
if [ -L ${BUILD_PROPERTIES_SYMLINK} ] ; then
info "Unlinking existing symbolic link: ${BUILD_PROPERTIES_SYMLINK}"
if ! unlink "${BUILD_PROPERTIES_SYMLINK}" ; then
warn "Failed to unlink $( readlink ${BUILD_PROPERTIES_SYMLINK} ) from ${BUILD_PROPERTIES_SYMLINK_DIR}"
fi
else
local backupFile="${BUILD_PROPERTIES_SYMLINK}.saved-by-quickstart.$(date +%Y-%m-%d-%H%M%S)"
info "Backing up your existing ~/.m2/**/${BUILD_PROPERTIES_BASENAME} file to ~/.m2/**/$( basename ${backupFile} )"
if ! mv "${BUILD_PROPERTIES_SYMLINK}" "${backupFile}" ; then
error "Failed to backup ${BUILD_PROPERTIES_SYMLINK}. Aborting properties file override. Please fix me!!"
return 1
fi
fi
fi
if ln -s "${BUILD_PROPERTIES_FILE}" "${BUILD_PROPERTIES_SYMLINK}" ; then
info "Override for ${BUILD_PROPERTIES_BASENAME} successful"
else
error "Override for ${BUILD_PROPERTIES_BASENAME} failed"
return 1
fi
}
function datawaveBuildSucceeded() {
local success=$( tail -n 7 "$DW_DATAWAVE_BUILD_STATUS_LOG" | grep "BUILD SUCCESS" )
if [ -z "${success}" ] ; then
return 1
fi
return 0
}
function buildDataWave() {
if ! mavenIsInstalled ; then
! mavenInstall && error "Maven install failed. Please correct" && return 1
fi
[[ "$1" == "--verbose" ]] && local verbose=true
! setBuildPropertyOverrides && error "Aborting DataWave build" && return 1
[ -f "${DW_DATAWAVE_BUILD_STATUS_LOG}" ] && rm -f "$DW_DATAWAVE_BUILD_STATUS_LOG"
info "DataWave build in progress: '${DW_DATAWAVE_BUILD_COMMAND}'"
info "Build status log: $DW_DATAWAVE_BUILD_STATUS_LOG"
if [ "${verbose}" == true ] ; then
( cd "${DW_DATAWAVE_SOURCE_DIR}" && eval "${DW_DATAWAVE_BUILD_COMMAND}" 2>&1 | tee ${DW_DATAWAVE_BUILD_STATUS_LOG} )
else
( cd "${DW_DATAWAVE_SOURCE_DIR}" && eval "${DW_DATAWAVE_BUILD_COMMAND}" &> ${DW_DATAWAVE_BUILD_STATUS_LOG} )
fi
if ! datawaveBuildSucceeded ; then
error "The build has FAILED! See $DW_DATAWAVE_BUILD_STATUS_LOG for details"
return 1
fi
info "DataWave build successful"
return 0
}
function getDataWaveTarball() {
# Looks for a DataWave tarball matching the specified pattern and, if found, sets the global 'tarball'
# variable to its basename for the caller as expected.
# If no tarball is found matching the specified pattern, then the DataWave build is kicked off
local tarballPattern="${1}"
tarball=""
# Check if the tarball already exists in the plugin directory.
local tarballPath="$( find "${DW_DATAWAVE_SERVICE_DIR}" -path "${tarballPattern}" -type f )"
if [ -f "${tarballPath}" ]; then
tarball="$( basename "${tarballPath}" )"
return 0;
fi
! buildDataWave --verbose && error "Please correct this issue before continuing" && return 1
# Build succeeded. Set global 'tarball' variable for the specified pattern and copy all tarballs into place
tarballPath="$( find "${DW_DATAWAVE_SOURCE_DIR}" -path "${tarballPattern}" -type f | tail -1 )"
[ -z "${tarballPath}" ] && error "Failed to find '${tarballPattern}' tar file after build" && return 1
tarball="$( basename "${tarballPath}" )"
# Current caller (ie, either bootstrap-web.sh or bootstrap-ingest.sh) only cares about current $tarball,
# but go ahead and copy both tarballs into datawave service dir to satisfy next caller as well
! copyDataWaveTarball "${DW_DATAWAVE_INGEST_TARBALL}" && error "Failed to copy DataWave Ingest tarball" && return 1
! copyDataWaveTarball "${DW_DATAWAVE_WEB_TARBALL}" && error "Failed to copy DataWave Web tarball" && return 1
return 0
}
function copyDataWaveTarball() {
local pattern="${1}"
local dwTarball="$( find "${DW_DATAWAVE_SOURCE_DIR}" -path "${pattern}" -type f | tail -1 )";
if [ -n "${dwTarball}" ] ; then
! cp "${dwTarball}" "${DW_DATAWAVE_SERVICE_DIR}" && error "Failed to copy '${dwTarball}'" && return 1
else
error "No tar file found matching '${pattern}'"
return 1
fi
return 0
}
# Bootstrap DW ingest and webservice components as needed
source "${DW_DATAWAVE_SERVICE_DIR}/bootstrap-ingest.sh"
source "${DW_DATAWAVE_SERVICE_DIR}/bootstrap-web.sh"
function datawaveIsRunning() {
datawaveIngestIsRunning && return 0
datawaveWebIsRunning && return 0
return 1
}
function datawaveStart() {
datawaveIngestStart
datawaveWebStart
}
function datawaveStop() {
datawaveIngestStop
datawaveWebStop
}
function datawaveStatus() {
datawaveIngestStatus
datawaveWebStatus
}
function datawaveIsInstalled() {
datawaveIngestIsInstalled && return 0
datawaveWebIsInstalled && return 0
return 1
}
function datawaveUninstall() {
datawaveIngestUninstall
datawaveWebUninstall
[[ "${1}" == "${DW_UNINSTALL_RM_BINARIES_FLAG_LONG}" || "${1}" == "${DW_UNINSTALL_RM_BINARIES_FLAG_SHORT}" ]] && rm -f "${DW_DATAWAVE_SERVICE_DIR}"/*.tar.gz
}
function datawaveInstall() {
datawaveIngestInstall
datawaveWebInstall
}
function datawavePrintenv() {
echo
echo "DataWave Environment"
echo
( set -o posix ; set ) | grep -E "DATAWAVE_|WILDFLY|JBOSS"
echo
}
function datawavePidList() {
datawaveIngestIsRunning
datawaveWebIsRunning
if [[ -n "${DW_DATAWAVE_WEB_PID_LIST}" || -n "${DW_DATAWAVE_INGEST_PID_LIST}" ]] ; then
echo "${DW_DATAWAVE_WEB_PID_LIST} ${DW_DATAWAVE_INGEST_PID_LIST}"
fi
}
function datawaveBuildDeploy() {
datawaveIsRunning && info "Stopping all DataWave services" && datawaveStop
datawaveIsInstalled && info "Uninstalling DataWave" && datawaveUninstall --remove-binaries
resetQuickstartEnvironment
export DW_REDEPLOY_IN_PROGRESS=true
datawaveInstall
export DW_REDEPLOY_IN_PROGRESS=false
}
function datawaveBuild() {
info "Building DataWave"
rm -f "${DW_DATAWAVE_SERVICE_DIR}"/datawave*.tar.gz
resetQuickstartEnvironment
}
Let's try Datawave out finally
So let's find some Wikipedia data based on page title, with a --verbose flag to see the cURL command in action with Datawave:
datawaveQuery --query "PAGE_TITLE:AccessibleComputing OR PAGE_TITLE:Anarchism" --verbose
Next let's grab TV show data from API.TVMAZE.COM(graph edge queries):
datawaveQuery --logic EdgeQuery --syntax JEXL --query "SOURCE == 'kevin bacon' && TYPE == 'TV_COSTARS'" --pagesize 30
Then, let's do our graph edge queries:
datawaveQuery --logic EdgeQuery --syntax JEXL --query "SOURCE == 'william shatner' && TYPE == 'TV_CHARACTERS'"
Let's try doing one more EdgeQuery:
datawaveQuery --logic EdgeQuery --syntax JEXL --query "SOURCE == 'westworld' && TYPE == 'TV_SHOW_CAST'" --pagesize 20
That was cool right? To formulate some of your own graph/edge queries run the following:
datawaveQuery --help
This will give you a broader scope on how Edges work.
Edges
One of the things I thought was, "EdgeQueryLogic in Datawave needs a better date range filter".
So to be clear, currently as it stands EdgeQueryLogic currently uses a column qualifier range filter to skip keys that are not within a specified date range. We need to be able to incorporate seeks such that the filter will skip over entries not within the date range. This is expected to be significantly faster when there are large gaps in sequence of edges for a source value that are not within the date range.
As you can see, this can come up short. An example subroutine that would help with Edge queries, would look like this:
private boolean seekToStartKey(Key topKey, String date) throws IOException
boolean seeked = false;
if (startDate != null && date.compareTo(startDate) < 0) {
// Date is before start date. Seek to same key with date set to start date
Key newKey = EdgeKeyUtil.getSeekToFutureKey(topKey, startDate);
PartialKey pk = EdgeKeyUtil.getSeekToFuturePartialKey();
fastReseek(newKey, pk);
} else if (endDate != null & date.compareTo(endDate) > 0) {
PartialKey part = EdgeKeyUtil.getSeekToNextKey();
fastReseek(topKey.followingKey(part), part);
seeked = true;
}
return seeked;
}
This now is getting into the core Datawave architecture. I've now helped you install it, run some queries on Wikipedia, and use an API for TV shows to demonstrate Edge querying.
Datawave Architecture
As you can imagine with things like ZooKeeper, Hadoop of course you'll be using things like MapReduce and Pig, below is the more-or-less the architecture of Datawave:
With this architecture, you might be thinking -- "What about run away queries?" Good question.
Your query can get into a state where the QueryIterator will not terminate. This is an edge case where yielding occurs, and the FinalDocument is returned as a result. This will get us into an infinite loop where the upon rebuilding the iterator, it will yield again and again.
Two things are required:
- The yield callback is checked in the
FinalDocumentIteratorand the yield is passed through appropriately. - The underlying iterator is no longer checked once the final document is returned.
This is just mainly another NB, this can happen, and if it does there's ways to do command jujitsu to get out of it.
Conclusion
We used EdgeQuerying got info from Wikipedia and gave it conditionals, and used some API's via REST to do some cool things, these are just some of the VERY few things Datawave can do, and that I've personally have done myself. I may do a part 2 series on this.
- Montana Mendy
Top comments (0)