ERROR: Segment connection failed: allocateWriterGang attempted to return a bad gang. (cdbgang.c:2591)

Question

Using Greenplum database version 3.2.3 on Solaris.

Step 1. Create a table.

CREATE TABLE ivdb.OPTION_PRICE (
    SecurityID integer NOT NULL,
    Date timestamp NOT NULL,
    Root char(5) NOT NULL,
    Suffix char(2) NOT NULL,
    Strike integer NOT NULL,
    Expiration timestamp NOT NULL,
    CallPut char(1),
    BestBid real NOT NULL,
    BestOffer real NOT NULL,
    LastTradeDate timestamp NULL,
    Volume integer NOT NULL,
    OpenInterest integer NOT NULL,
    SpecialSettlement char(1) DEFAULT '0',
    ImpliedVolatility real NOT NULL,
    Delta real NOT NULL,
    Gamma real NOT NULL,
    Vega real NOT NULL,
    Theta real NOT NULL,
    OptionID integer NOT NULL,
    Adjustmentfactor integer DEFAULT 1 NOT NULL,

    CONSTRAINT PK_OPTION_PRICE PRIMARY KEY (Date, Root, Suffix))

    PARTITION BY RANGE (Date) (
        START (timestamp '01/01/1996') INCLUSIVE
        END (timestamp '01/01/2020') EXCLUSIVE
        EVERY (INTERVAL '1 month')); 

Step 2: Insert data from another table. (This one is plain vanilla, not partitioned, no constraints. It has 564,392,723 rows.)

INSERT INTO OPTION_PRICE SELECT * FROM casey_option_price;

Results:

-- Executing query:

INSERT INTO OPTION_PRICE SELECT * FROM casey_option_price;
NOTICE: Releasing gangs to finish aborting the transaction.


ERROR: Segment connection failed: allocateWriterGang attempted to return a bad gang. (cdbgang.c:2591)

********** Error **********

ERROR: Segment connection failed: allocateWriterGang attempted to return a bad gang. (cdbgang.c:2591)
SQL state: XX000

The bad gang things brings the whole show to a halt, need to restart database to get things cleaned up again.

Haven't found much on the web, have a helpdesk ticket open with greenplum, thought I'd float it out here as well. Will come back with a solution if I get one before you do.

Sorry, not enough rep to tag with "greenplum".

Answer 1

This error was due to a hardware problem. A hard drive failed and for some reason the RAID didn't cover us correctly.

"bad gang" means "check your hardware" to me now.

A related (or maybe the real one) problem: Check your gp_vmem_protect_limit setting. Ours was too high, and I was using up all of the machine's swap space in my query.

Answer 2

The "gang was disconnected" is a symptom which indicates one or more primary segments' worker processes abort abnormally. The possible causes vary. EG, max_connections are reached on one segment; primary segments down due to timeout; Postgresql Processes are killed; segment server NIC issue; File system is full on segments; etc.

I suggest you open cases to GP support team with below info:

1. master logs
2. related segment logs
3. gp_segment_configuration output
4. select * from gp_configuration_history order by 1 desc;
5. /var/log/messages on related segment servers
6. df -h on segments
7. Any change you may think related.