elasticsearch - Data Store with multiple shard keys -
i've been researching several different data store technologies can used storing huge amounts of semi-structured logs people search through later. i've looked @ cassandra, riak, , elastic search far, , seems elastic search offers closest fit i'm interested in (largely because indexes transparently). however, there's 1 feature i'm interested in seems escape them all, , wondering if there data store feature.
what i'm thinking ability transparent shard on multiple key. clear, i'm not talking using composite key sharding. mean if had table sharded user_id
, time_of_creation
, , ip_address
, , inserted row, 3 copies of row created, each 1 in different cluster that's sharded different key (or maybe somehow in same cluster. important part data duplicated). , when wanted query table later, data store transparently choose cluster use.
in articles i've read cassandra, people recommend doing this, it's manual process in @ least 3 ways:
- for insertion, have insert each table yourself.
- when comes querying, have figure out table want query (you need pick 1 uses right cluster key).
- and, if ever want add key shard on, have write routine existing data new table.
although using cassandra example, believe situation riak , elastic search similar. understand data store offered ability have make huge trades so. updating/deleting might no longer possible (or have extremely poor performance), , consistency suffer. but, it's set of trades find acceptable when dealing logs, wondering if familiar technology offers feature.
Comments
Post a Comment